{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 为什么Bagging能改进模型性能？（10分） \n",
    "2. 随机森林中随机体现在哪些方面？（10分） \n",
    "3. 随机森林和GBDT的基学习器都是决策树。请问这两种模型中的决策树有什么不同。（10分） \n",
    "4. 请简述LightGBM的训练速度问什么比XGBoost快。（30分） \n",
    "5. 采用tfidf特征，采用LightGBM（gbdt）完成Otto商品分类，尽可能将超参数调到最优，并用该模型对测试数据进行测试，提交Kaggle网站，提交排名。（20分） \n",
    "6. 采用tfidf特征，采用LightGBM（goss）完成Otto商品分类，尽可能将超参数调到最优，并用该模型对测试数据进行测试，提交Kaggle网站，提交排名。（20分） "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.Bagging是一种降低算法方差的方式：采用了平均多个模型的预测。其适合对偏差低、方差高的模型进行融合，达到减小方差的目的。\n",
    "\n",
    "2.随机森林即Bagging多棵决策树。随机森林随机选取一部分特征，随机选取一部分样本，来降低树的相关性。\n",
    "随机森林的随机性主要体现在两个方面：\n",
    "   数据集的随机选取：从原始的数据集中采取有放回的抽样（bagging），构造子数据集，子数据集的数据量是和原始数据集相同的。不同子数据集的元素可以重复，同一个子数据集中的元素也可以重复。\n",
    "   待选特征的随机选取：与数据集的随机选取类似，随机森林中的子树的每一个分裂过程并未用到所有的待选特征，而是从所有的待选特征中随机选取一定的特征，之后再在随机选取的特征中选取最优的特征。\n",
    "   \n",
    "3.随机森林是一个用随机方式建立的，包含多个决策树的集成分类器。其输出的类别由各个树投票而定（如果是回归树则取平均）。GBDT即梯度提升决策树，GB是以决策树为基学习器的迭代算法，GBDT里的决策树都是回归树。\n",
    "组成随机森林的树可以是分类树，也可以是回归树；而GBDT只能由回归树组成。\n",
    "组成随机森林的树可以并行生成；而GBDT只能是串行生成。\n",
    "对于最终的输出结果而言，随机森林采用多数投票等；而GBDT则是将所有结果累加起来，或者加权累加起来。\n",
    "\n",
    "4.采用了GOSS：通过对样本进行下采样减少样本；采用了EFB：通过互斥特征捆绑，减少特征数目。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import lightgbm as lgbm\n",
    "from lightgbm.sklearn import LGBMClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "train_data_path1=open(r'C:\\Users\\dell\\Downloads\\201913164947853_97888\\3代码类素材\\data\\Otto_FE_train_org.csv')\n",
    "train_data_path2=open(r'C:\\Users\\dell\\Downloads\\201913164947853_97888\\3代码类素材\\data\\Otto_FE_train_tfidf.csv')\n",
    "train1=pd.read_csv(train_data_path1)\n",
    "train2=pd.read_csv(train_data_path2)\n",
    "train2 = train2.drop([\"id\",\"target\"], axis=1)\n",
    "train =  pd.concat([train1, train2], axis = 1, ignore_index=False)\n",
    "train.head()\n",
    "\n",
    "del train1\n",
    "del train2\n",
    "#train1.head()\n",
    "#train2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 将类别字符串变成数字\n",
    "y_train = train['target'] #形式为Class_x\n",
    "y_train = y_train.map(lambda s: s[6:])\n",
    "y_train = y_train.map(lambda s: int(s) - 1)#将类别的形式由Class_x变为0-8之间的整数\n",
    "\n",
    "X_train = train.drop([\"id\", \"target\"], axis=1)\n",
    "\n",
    "#保存特征名字以备后用（可视化）\n",
    "feat_names = X_train.columns \n",
    "\n",
    "from scipy.sparse import csr_matrix\n",
    "X_train = csr_matrix(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " 以下为LightGBM超参数调优"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "MAX_ROUNDS = 10000\n",
    "\n",
    "from sklearn.model_selection import StratifiedKFold#交叉验证分组\n",
    "kfold = StratifiedKFold(n_splits=3, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#直接调用lightgbm内嵌的交叉验证(cv)，对连续的n_estimators参数进行快速交叉验证\n",
    "def get_n_estimators(params , X_train , y_train , early_stopping_rounds=10):\n",
    "    lgbm_params = params.copy()\n",
    "    lgbm_params['num_class'] = 9\n",
    "     \n",
    "    lgbmtrain = lgbm.Dataset(X_train , y_train )\n",
    "     \n",
    "    #num_boost_round为弱分类器数目，下面的代码参数里因为已经设置了early_stopping_rounds\n",
    "    #即性能未提升的次数超过过早停止设置的数值，则停止训练\n",
    "    cv_result = lgbm.cv(lgbm_params , lgbmtrain , num_boost_round=MAX_ROUNDS , nfold=3,  metrics='multi_logloss' , early_stopping_rounds=early_stopping_rounds,seed=3 )\n",
    "     \n",
    "    print('best n_estimators:' , len(cv_result['multi_logloss-mean']))\n",
    "    print('best cv score:' , cv_result['multi_logloss-mean'][-1])\n",
    "     \n",
    "    return len(cv_result['multi_logloss-mean'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best n_estimators: 428\n",
      "best cv score: 0.475238589648195\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'num_leaves': 60,\n",
    "          'max_depth': 6,\n",
    "          'max_bin': 127, #2^6\n",
    "          'subsample': 0.7,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "\n",
    "n_estimators_1 = get_n_estimators(params , X_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done   8 out of  12 | elapsed: 12.7min remaining:  6.3min\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  12 | elapsed: 18.1min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "       error_score='raise',\n",
       "       estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt', class_weight=None,\n",
       "        colsample_bytree=0.7, importance_type='split', learning_rate=0.1,\n",
       "        max_bin=127, max_depth=7, min_child_samples=20,\n",
       "        min_child_weight=0.001, min_split_gain=0.0, n_estimators=428,\n",
       "        n_jobs=4, num_class=9, num_leaves=31, objective='multiclass',\n",
       "        random_state=None, reg_alpha=0.0, reg_lambda=0.0, silent=False,\n",
       "        subsample=0.7, subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid=True, n_jobs=4,\n",
       "       param_grid={'num_leaves': range(50, 90, 10)},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#调整num_leaves,并相应调整max_bin\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'max_bin': 127, \n",
    "          'subsample': 0.7,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "num_leaves_s = range(50,90,10)\n",
    "tuned_parameters = dict( num_leaves = num_leaves_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4, param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4738546380380549\n",
      "{'colsample_bytree': 0.4}\n"
     ]
    }
   ],
   "source": [
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAELCAYAAADkyZC4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xd4VGX2wPHvSUIghFACAemhN4FA\nAoqr2FBB17oqIiDYULHr6qK7P3fXXRVxrSh2QAXEvqJSLQg2IKEHkhAgQKgpEEIKaef3xwyILAMx\nZHKnnM/zzMPcO3fuPS83M2fue997rqgqxhhjTHULcToAY4wxgckSjDHGGK+wBGOMMcYrLMEYY4zx\nCkswxhhjvMISjDHGGK+wBGOMMcYrLMEYY4zxCkswxhhjvCLM6QCc1KRJE42NjXU6DGOM8StJSUnZ\nqhpzouWCOsHExsaSmJjodBjGGONXRGRLZZazLjJjjDFeYQnGGGOMV1iCMcYY4xWWYIwxxniFJRhj\njDFeYQnGGGOMV1iCMcYY4xWWYIwxJoioKm8s2sju/cVe35YlGGOMCSILU7N4cnYKc9fu8vq2LMEY\nY0yQKK9Qxs9JIbZxXYb1b+P17VmCMcaYIPHJ8kxSd+fz0EVdCQ/z/te/JRhjjAkCRSXlPDc/jd6t\nG3Jxz1NqZJuWYIwxJghM/nEzu/YX8+iQrohIjWzTEowxxgS43IISXlu4kUHdmnFa+8Y1tl1LMMYY\nE+AmfruBgpIy/jK4S41u1xKMMcYEsC05BUz7ZQtD+7WmU7OoGt22JRhjjAlgz8xLJSwkhPsGda7x\nbVuCMcaYALVq2z6+XL2TW89qR7P6dWp8+5ZgjDEmAKkqT85eT+PIcMac3cGRGCzBGGNMAPoudQ9L\nNudy76BO1Ksd5kgMlmCMMSbAlJVX8NTsFNo1iayRkjCeWIIxxpgA88nyTDbsOcDDF3WhVqhzX/OW\nYIwxJoAUlZTz3II0+rRpyOBTa6YkjCeWYIwxJoBM/nEzu/cf5NGLu9VYSRhPLMEYY0yAyDlwkFcX\nbuSC7s3oFxvtdDiWYIwxJlBM/DadotJy/jK4q9OhAJZgjDEmIGRk/1oSpmPTek6HA1iCMcaYgPDM\n/FTCw0K4b1Anp0M5zBKMMcb4uRVb9/LV6p3cclZ7mkbVfEkYTyzBGGOMH1NVnpqTQpN64YwZ2N7p\ncH7DEowxxvixb9bvYenmXO4d1NmxkjCeWIIxxhg/VVZewdNzU2jfJJLr+rV2Opz/YQnGGGP81MdJ\n7pIwg50tCeOJ70VkjDHmhApLynhuQRp92zTkoh7OloTxxBKMMcb4obcXb2ZPvm+UhPHEEowxxviZ\n7AMHeX3RJi7q0YwEHygJ44klGGOM8TMTv9lAUWk5D/tISRhPLMEYY4wf2ZxdwPQlW7muX2s6xPhG\nSRhPLMEYY4wfeWZeCuFhIdzrQyVhPPFqghGRwSKSKiLpIjLuOMtdLSIqIgnu6eEisvKIR4WIxLlf\nGyoiq0UkWUQmHLGOB0Rknfu1b0SkrTfbZowxNW351r3MXrOLMQN9qySMJ15LMCISCrwCDAG6A8NE\npPsxlosC7gGWHJqnqtNVNU5V44CRQIaqrhSRxsAzwPmq2gNoJiLnu9+2AkhQ1V7Ax8AEjDEmQKgq\n42en0KRebW49y7dKwnjizSOY/kC6qm5S1RJgJnD5MZb7F65kUOxhPcOA993P2wNpqprlnv4a+BOA\nqn6nqoXu+b8ArU6+CcYY4xu+Xr+HpRm53DeoE5E+VhLGE28mmJbAtiOmM93zDhORPkBrVf3yOOsZ\nyq8JJh3oKiKxIhIGXAEcqz7CzcCcqgZujDG+pKy8gvFz1tM+JpKhPlgSxhNvpsFjXfmjh18UCQGe\nB0Z7XIHIaUChqq4FUNW9InIH8AFQAfyE66jmyPeMABKAsz2scwwwBqBNmzaVb40xxjjkw8RMNmYV\n8PrIeJ8sCeOJNyPN5LdHF62AHUdMRwGnAgtFJAM4HZh16ES/23X8evQCgKp+oaqnqeoAIBXYcOg1\nERkE/BW4TFUPHisoVX1DVRNUNSEmJqbKjTPGmJpQWFLG81+nkdC2ERd2b+Z0OL+LN49glgGdRKQd\nsB1Xsrj+0Iuqmgc0OTQtIguBP6tqons6BLgGGHjkSkWkqaruEZFGwFjgWvf8PsDrwGBV3ePFdhlj\nTI15a/FmsvIP8tqIeJ8tCeOJ1xKMqpaJyF3APCAUmKyqySLyOJCoqrNOsIqBQKaqbjpq/osi0tv9\n/HFVTXM/fwaoB3zk3glbVfWyammMMcY4ICv/IK9/v5HBPU4hvm0jp8P53bw6FEFVZwOzj5r3mIdl\nzzlqeiGubrOjlxvm4f2DqhqnMcb4ope+2UBxWQUPD+7idChV4j9ni4wxJohsyjrAjKVbub5/G9r7\neEkYTyzBGGOMD3pmXip1wkK453zfLwnjiSUYY4zxMUlb9jJn7S7GDOxATFRtp8OpMkswxhjjQ1SV\np2avJyaqNrec1c7pcE6KJRhjjPEh89ftJnHLXu4f1NlvSsJ4YgnGGGN8RFl5BU/PTaFDTCTXJvh/\nOUVLMMYY4yM+SNzGpqwCxg3pRpgflYTxxP9bYIwxAaDgYBnPL9hAv9hGDOrW1OlwqoUlGGOM8QFv\nLt5E9oGDPHJxN78rCeOJJRhjjHHYnvxi3li0iYt7nkLfNv5XEsYTSzDGGOOwl77ZQElZBQ9d1NXp\nUKqVJRhjjHHQxqwDvL90G9ef1oZ2TSKdDqdaWYIxxhgHTZib4vclYTyxBGOMMQ5JzMhlXvJubj+7\nA03q+W9JGE8swRhjjANUlafmpNA0qjY3+3lJGE8swRhjjAPmJe8macteHrigM3XD/bskjCeWYIwx\npoaVllcwYW4KHZvW4+p4/y8J44klGGOMqWEzl21jU3YB4wZ3DYiSMJ4EbsuMMcYHHThYxotfp9G/\nXTTnB0hJGE8Cs+PPGGN81JuLNpF9oIS3RgVOSRhP7AjGGGNqyJ79xby5eBOX9GxOXOuGTofjdZZg\njDGmhrxwuCRMF6dDqRGWYIwxpgak7znAB8u2MeL0tsQGWEkYTyzBGGNMDZgwN4WIWqHcfV5Hp0Op\nMZZgjDHGy5Zl5DJ/3W5uP7s9jQOwJIwnlmCMMcaLVJUnZ6+nWf3a3Hxme6fDqVGWYIwxxovmrt3F\niq37eOCCzkSEhzodTo2yBGMCQkWF8tr3G1m7Pc/pUIw5rLS8ggnzUuncrB5/6hu4JWE8sQRjAsLE\nb9MZPyeFGyYvZUtOgdPhGAPAzKVb2ZxdwLghgV0SxpPga7EJON+l7uGFb9IY1K0pFarcOHUZ+wpL\nnA7LBLkDB8t44esNnNYumnO7BHZJGE8swRi/ti23kPtmrqRLsygmDuvLGyMTyMwt4rb3kigpq3A6\nPBPE3vh+IzkFJTxyceCXhPHEEozxW8Wl5dw+LYkKVV4fGU9EeCj920XzzDW9WLI5l3GfrkZVnQ7T\nBCFXSZjN/LFXcJSE8cSKXRq/pKr87b9rSd6xn7dHJdC28a9XRl8e15ItOYU8tyCN2MaRAXmvc+Pb\nnv96A2UVwVMSxhNLMMYvvb90Gx8nZXLPeR05v1uz/3n97vM6kpFTwHML0mgTXZcr+rR0IEoTjNL3\n5PPBsq3cMCD2Nz98gpElGON3Vm7bxz9mJTOwcwz3Dup8zGVEhPFX9WLHviIe/ng1LRpG0L9ddA1H\naoLR+DmpRIaHBVVJGE/sHIzxKzkHDjJ2WhIxUbV5cWgcoSGeT56Gh4Xw2oh4WkVHMOa9RDZn2/Bl\n411LNuXw9frd3H5Oh6AqCeOJVxOMiAwWkVQRSReRccdZ7moRURFJcE8PF5GVRzwqRCTO/dpQEVkt\nIskiMuGIddQWkQ/c21oiIrHebJupeeUVyj0zV5BdUMJrI+JpFBl+wvc0rBvOlNH9CBHhxilL2Vtg\nw5eNd6gqT81J4ZT6dbjpD+2cDscneC3BiEgo8AowBOgODBOR7sdYLgq4B1hyaJ6qTlfVOFWNA0YC\nGaq6UkQaA88A56tqD6CZiJzvftvNwF5V7Qg8DzztrbYZZzw7P5Uf03P49+Wn0rNVg0q/r23jSN68\nIZ4decWMeS+Rg2XlXozSBKs5a3excts+Hrgw+ErCeOLNI5j+QLqqblLVEmAmcPkxlvsXMAEo9rCe\nYcD77uftgTRVzXJPfw38yf38cuAd9/OPgfMlWAefB6B5ybuYtHAjw/q35tp+rX/3++PbRvPsNb1Z\nlrGXhz+24cumepWUVTBhbgpdmkUFZUkYT06YYESkg4jUdj8/R0TuEZHKDOxuCWw7YjrTPe/IdfcB\nWqvql8dZz1B+TTDpQFcRiRWRMOAK4NC3zeHtqWoZkAc0rkScxsdtyjrAnz9cRa9WDfj7pT2qvJ5L\ne7fgoYu68PnKHTz/9YZqjNAEu/eXbiUjp5BxQ7oe97xgsKnMEcwnQLmIdATeBtoBMyrxvmP9Lx/+\n2SgiIbi6sh70uAKR04BCVV0LoKp7gTuAD4DFQAZQVpntHbHOMSKSKCKJWVlZx3iL8SWFJWXcPi2J\nsFBh0vC+1Kl1cl0PY8/pwNCE1rz0zQY+TsqspihNMMsvLuXFbzYwoH1jzukS43Q4PqUyCabCfURw\nJfCCqt4PNK/E+zL59egCoBWw44jpKOBUYKGIZACnA7MOneh3u45fj14AUNUvVPU0VR0ApAKHfooe\n3p776KYBkHt0UKr6hqomqGpCTIz9MfgyVWXcJ2vYsOcALw3rQ6tGdU96nSLCv688lT90bMwjn67m\np43Z1RCpCWZvLNpEbkEJj1zcNWhLwnhSmQRTKiLDgFHAoa6sWpV43zKgk4i0E5FwXMli1qEXVTVP\nVZuoaqyqxgK/AJepaiIcPsK5Bte5m8NEpKn730bAWOAt90uz3DECXA18q9bR7tem/pTBrFU7+POF\nXTirU/X9GKgVGsKk4fHENo7k9veSSN9zoNrWbYLL7v3FvLl4E5f2bkGvVsFbEsaTyiSYG4EBwBOq\nullE2gHTTvQm91HPXcA8YD3woaomi8jjInJZJbY7EMhU1U1HzX9RRNYBPwLjVTXNPf9toLGIpAMP\nAB6HRRvftywjlye+Ws+gbs244+wO1b7+BhG1mDy6H+FhIdw4dSk5Bw5W+zZM4Ht+QRrlFcpDFwZ3\nSRhP5Pf8yHcfNbRW1dXeC6nmJCQkaGJiotNhmKPs2V/MJRN/IDI8lM/vOpMGEZU5YK6aldv2MfT1\nn+nRoj4zbj39pM/xmOCRtjufwS8sYvQZ7Xjs0v+5AiOgiUiSqiacaLnKjCJbKCL1RSQaWAVMEZHn\nqiNIY45WWl7BXTNWkF9cymsj472aXADiWjfkhaFxrNi2jwc/WkVFhfWqmsp5ek4KkbWtJMzxVKaL\nrIGq7geuAqaoajwwyLthmWA1fk4KSzNyGX9VL7qeUr9GtjmkZ3MeGdKVr1bv5D/zU2tkm8a//bIp\nh29S9jD2nI6VqigRrCpT7DJMRJoD1wJ/9XI8Joh9sWoHb/+wmdFnxNZ49eNbz2pPRk4hkxZupG3j\nugzt16ZGt2/8h6ry1Oz1NG9Qhxv/EOt0OD6tMkcwj+M6Ub9RVZeJSHt+HRpsTLVI253PXz5ZTXzb\nRjx6cbca376I8PhlPRjYOYa/fraWHzbY8GVzbF+t2cmqzDweuKCznbM7gRMmGFX9SFV7qeod7ulN\nqvqnE73PmMrKLy7l9veSqBsexqThfQkPc6bId1hoCK9c34eOTetxx7Qk0nbnOxKH8V2ukjCpdD0l\niqusJMwJVeYkfysR+UxE9ojIbhH5RETsf9ZUC1XloY9WsyW3kJev70Oz+nUcjSeqTi3eHt2POuGh\n3DhlGVn5NnzZ/GrGki1szbWSMJVVmZ+KU3BdxNgCV72vL9zzjDlpbyzaxNzkXTwypCunt/eN0nEt\nG0YweVQ/cgtKuOXdRIpKrPqygf3Fpbz0bTpndGjM2Z2tCkhlVCbBxKjqFFUtcz+mAva/a07aT+nZ\nPD03hUt6NufmM33r/hk9WzXgxeviWJ25j/s/WGnDlw2vf7/RVRJmSDcrCVNJlUkw2SIyQkRC3Y8R\nQI63AzOBbWdeEXe/v4L2MfV4+upePvmBvbDHKfztku7MTd7F03NTnA7HOGhnXhFvLd7M5XEtfte9\niIJdZYYp3wS8jKvysQI/4SofY0yVHCwr545pyykuLee1EfHUq12ZP0Nn3PSHWLbkFPD6ok20aVyX\n4ae1dTok44DnF6ShCn+2kjC/S2VGkW1V1ctUNUZVm6rqFbguujSmSv795XpWbtvHf67pTcem9ZwO\n57hEhMf+2J1zu8Tw2OfJfJ9mt3gINqm78vk4KZMbBrSldfTJV/QOJlUdD/pAtUZhgsYnSZm898sW\nbhvYniE9K3PXB+eFhYYw8fq+dG4WxZ3Tl5Oya7/TIZka9PRcV0mYO8+1kjC/V1UTjO91mBufl7wj\nj0c/W8Pp7aN56CL/6mqoVzuMyaMTiKwdyk1TlrF7v6c7fJtA8tPGbL5N2cOd51pJmKqoaoKxITXm\nd8krLOWOactpVDecicP6EhbqzMWUJ6N5gwgmj+7HvqJSbn5nGYUlZSd+k/FbFRXK+DkptGhQh9Fn\nxDodjl/y+CkXkXwR2X+MRz6ua2KMqZSKCuX+D1eyM6+IV4b3JSaqttMhVVmPFg14+fo+rNuxn3ve\nX0m5DV8OWF+t2cnqzDwevLCLlYSpIo8JRlWjVLX+MR5Rquq7w36Mz3n5u3S+TdnDY3/sTnzbRk6H\nc9LO69qMf1zWg6/X7+aJr9Y7HY7xgoNl5UyYl0LXU6JqvPBqILFEYbxqYeoenv86jav6tGTE6YEz\nxPeGAbFkZBcy+cfNtG1cl1HWhRJQpv+ylW25RbxzU38rCXMSLMEYr9mWW8i9M1fSpVkUT1zZ0ycv\npjwZf72kG1tzC/nnF8m0jo7gvK7NnA7JVIO8olImfruBMzs2YWCnJk6H49f870yr8QvFpeXcMT2J\nClVeHxlPRHjg9WGHhggvDYuje4v63DVjBck78pwOyVSD177fyN7CUsYN6RpwP4pqmiUYU+1Ulf/7\n71rWbt/PC0PjaNs40umQvKZueBhvj+pHw4ha3DR1GTvzipwOyZyEnXlFTP5hM1f2acmpLa0kzMmq\nTLn+Y40m2+Yu4d++JoI0/mXmsm18lJTJPed15Pxugd9t1Kx+Hd4e3Y+Cg+XcPDWRAwdt+LK/em6+\nqyTMAxd0djqUgFCZI5jngIdwlepvBfwZeBOYCUz2XmjGH63ato+/f57MwM4x3DsoeD6k3ZrX55Xh\nfUndnc/dM5ZTVl7hdEjmd0rZtZ+Pl2cy6gwrCVNdKpNgBqvq66qar6r7VfUN4GJV/QDw/zGnptrk\nFpRwx7QkYqJq8+LQuKAbfXN25xgev7wH36Vm8fiX61C1a2T8yfg5KURZSZhqVZkEUyEi14pIiPtx\n7RGv2SfIAFBeodzz/gqyC0p4bUR80JbVGH5aW8YMbM+7P29hyo8ZTodjKumn9GwWpmZx13kdaVg3\nOP92vaEyCWY4MBLY436MBEaISARwlxdjM37kuQWp/JCezb8vPzXo75cxbnBXBvc4hX99tY75ybuc\nDsecQEWF8tScFFo2jOCGAbFOhxNQKlOuf5OqXqqqTdyPS1U1XVWLVPWHmgjS+Lb5ybt45buNDOvf\nmmv7tXY6HMeFhAjPD42jV8sG3DtzJWsybfiyL/ti9Q7WbM/jwQs7W0mYalaZUWSt3CPG9ojIbhH5\nRERa1URwxvdtzi7gwQ9X0bNlA/5+aQ+nw/EZEeGhvDkqgejIcG56Zxnb99nwZV90sKycZ+al0q15\nfa6Is5Iw1a0yXWRTgFm4Cly2BL5wzzNBrrCkjNvfSyI0VHh1RF/79XeUplF1mHJjP4pLyrl56jLy\ni0udDskc5b2ft5C5t4hHL+5KSJANSqkJlUkwMao6RVXL3I+pQIyX4zI+TlV55NM1pO3J56Xr+tCq\nkQ3rPJbOzaJ4dUQ86XsOcOeMFZTa8GWfkVdUysvfpXNWpyac1cm+0ryhMgkmW0RGiEio+zECyPF2\nYMa3vfNTBp+v3MGDF3RmYGf7cB7PmZ2a8MSVp7IoLYu/z0q24cs+4tWFG8krcpWEMd5RmQRzE3At\nsAvYCVwN3OjNoIxvS8zI5d9frWdQt6aMPceuGaiMof3aMPacDsxYspU3F29yOpygt31fEZN/3MyV\ncS3p0SK4Rz160wmrKavqVuCyI+eJyH3AC94KyviuPfnFjJ2+nJaNInj22jjrt/4d/nxhF7bkFvLk\n7BRaN6rLkJ7NnQ4paD03Pw2ABy4MnmoTTqhqscsHqjUK4xdKyyu4a8YK9heX8tqIeBpE1HI6JL8S\nEiI8e01v+rZpyH0frGTF1r1OhxSU1u/cz6crMrnxjFg7d+hlVU0w9rM1CD09J4Wlm3N56qqedGte\n3+lw/FKdWqG8eUMCzerX4dZ3E9mWW+h0SEFn/JwU6tepZd27NaCqCcbOUgaZL1fv4K0fNjNqQFuu\n7GOXQZ2MxvVqM3l0P0rKKrhp6jLyimz4ck35YUM236dlcde5HWlQ147Avc1jgvFQpn+/iOTjuibG\nBIkNu/N5+OPV9G3TkL9e0t3pcAJCx6b1eH1kAhk5BYydnmTDl2uAqyTMelo2jGDkgMC5fbcv85hg\nVDVKVesf4xGlqnar5SCRX1zKbdOSqBseyqTh8YSH2T3qqsuADo156qpe/Jiew98+W2vDl71s1qod\nJO/Yz0MXdbGLgmuIV78tRGSwiKSKSLqIjDvOcleLiIpIgnt6uIisPOJRISJx7teGicgaEVktInNF\npIl7fpyI/OJePlFE+nuzbcFAVXnoo9VsySlk4rC+nNKgjtMhBZyr41txz3kd+SBxG69+v9HpcALW\noZIwPVrU57Le1gFTU7yWYEQkFHgFGAJ0B4aJyP/0r4hIFHAPsOTQPFWdrqpxqhqHq3pzhqquFJEw\n4EXgXFXtBazm14rOE4B/ut/zmHvanIQ3Fm1ibvIuxg3uyoAOjZ0OJ2Ddf0FnLo9rwYS5qXy5eofT\n4QSk937ewvZ9RTwypJsNra9B3jyC6Q+ku6sxl+C6A+blx1juX7iSQbGH9QwD3nc/F/cjUkQEqA8c\n+kSqexqgwRHzTRX8tDGbp+emcHHPU7jlrHZOhxPQRISn/9SLfrGNeODDVSRtseHL1SmvsJSJ36Yz\nsHMMZ3Zq4nQ4QcWbCaYlsO2I6Uz3vMNEpA/QWlW/PM56huJOMKpaCtwBrMGVQLoDb7uXuw94RkS2\nAf8BHjnWykRkjLsLLTErK+t3NyoY7Mwr4u4ZK2jXJJIJV/fGlcuNN9WpFcrrIxNo0cA1fHlrjg1f\nri6TFqazv7iUcYOtJExN82aCOda30uGzmCISAjwPPOhxBSKnAYWqutY9XQtXgumDayTban5NJHcA\n96tqa+B+fk08vw1A9Q1VTVDVhJgYq6F1tJKyCsZOX05xaTmvj4ynXm0bz1FToiPDmXJjfypUGT11\nKXmFNnz5ZG3fV8SUnzK4qk8rurewa7dqmjcTTCZw5N2nWvHbbqso4FRgoYhkAKcDsw6d6He7jl+7\nxwDiAFR1o7qG3HwInOF+bRTwqfv5R7i66Mzv9O+v1rFi6z6euaY3HZtGOR1O0GnXJJI3RiaQmVvE\nbdMSKSmz4csn49n5qYCVhHGKNxPMMqCTiLQTkXBcyWLWoRdVNc99h8xYVY0FfgEuU9VEOHyEcw2u\nczeHbAe6i8ihQ48LgPXu5zuAs93PzwM2eKdZgevT5Zm8+/MWxgxsz8VWJ8sx/dtFM+HqXvyyKZdH\nPl1jw5erKHlHHp+t2M6Nf4ilZcMIp8MJSl7r/1DVMhG5C5gHhAKTVTVZRB4HElV11vHXwEAgU1UP\nl55V1R0i8k9gkYiUAluA0e6XbwVedI80KwbGVG+LAtu6Hft59LM1nN4+mocv6uJ0OEHvij4t2ZJT\nyPNfp9G2cV3uOb+T0yH5nfFzUmgQYSVhnOTVDnZVnQ3MPmreYx6WPeeo6YW4us2OXu414LVjzP8B\niK96tMErr7CU26cl0SCiFhOH9SUs1C6m9AX3nN+RLTkFPLcgjTbRdbmij93St7IWb8hi8YZs/nZJ\nNyvK6iA7gxvkKiqUBz5cyc68ImaOGUBMVG2nQzJuIsJTf+rJ9n1FPPzxalo0jKB/u2inw/J5FRXK\nU7NTaNXISsI4zX6qBrlXvkvnm5Q9/N8fuxPftpHT4Zij1A4L5fWR8bSKjmDMe4lszi5wOiSf9/mq\n7azb6SoJUzvMSsI4yRJMEPs+LYvnvk7jyj4tGXm6/dLzVQ3rhjNldD9CRLhxylL2FpQ4HZLPKi4t\n5z/z0ji1ZX0u7WUlYZxmCSZIbcst5N6ZK+jSLIonr+xpF1P6uLaNI3nzhnh25BUz5r1EDpaVOx2S\nT3r35wy27yviUSsJ4xMswQSh4tJy7pieRHmF8tqIeCLCrRvBH8S3jebZa3qzLGMvD3+82oYvH2Vf\nYQkvf5vOOV1iOKOjlYTxBXaSPwj9/fNk1m7fz1s3JBDbJNLpcMzvcGnvFmzNLeSZeam0bRzJAxfY\nBYSHTFq4kfyDZfzFSsL4DEswQWbm0q18kLiNu8/ryKDuzZwOx1TB2HM6sCWngJe+2UDb6Lr8Kd7u\nMLott5CpP2bwp76t7HbePsQSTBBZtW0fj32ezFmdmnDfIPvl669EhCeudA1fHvepa/hysN9O4bkF\naYhgR3Q+xs7BBIncghLGTl9OTFRtXrquD6F2AtSv1QoNYdLweNo2juS29xJJ33PA6ZAcs3a7qyTM\nTWe2o4WVhPEplmCCQHmFcu/MFWQdOMirI/rSKDLc6ZBMNWgQUYspo/sRHhbCTVOXkXPgoNMhOeLp\nuSk0rFuL28/u4HQo5iiWYILA8wvSWLwhm39d3oNerRo6HY6pRq2j6/LmDQns3l/Mre8mUlwaXMOX\nF6W5SsLcfV4nKwnjgyzBBLgF63bz8nfpXNevNUP7tXE6HOMFfdo04oWhcSzfuo8/f7SKiorgGL5c\nXqE8NSeF1tERjDjd/rZ9kSWYALY5u4AHPlhJz5YN+MdlPZwOx3jRkJ7NeWRIV75cvZNnF6Q6HU6N\n+O+K7azfuZ+HLupqJWF8lI3Uy267AAATfklEQVQiC1CFJWXcMS2J0FDh1RF9qVPLPoCBbszA9mTk\nFPLKdxtpGx3Jtf1an/hNfqq4tJxn56fSq1UD/mj3LvJZdgQTgFSVRz5dQ+rufF66rg+tGtV1OiRT\nA0SExy/vwVmdmvDoZ2v4MT3b6ZC85p2fMtiRV8y4IV2tJIwPswQTgN79eQufr9zBgxd0ZmDnmBO/\nwQSMWqEhvDK8Lx1i6nH7tCQ27M53OqRqt7eghJe/S+fcLjGc0cFKwvgySzABJmlLLv/6ch2DujW1\nO/kFqfp1ajH5xn7UqRXK6CnLyMoPrOHLr3yXTsHBMsYN6eZ0KOYELMEEkD35xYydvpyWjSJ49to4\n6zoIYi0bRvD2qARyC0q45d1EikoCY/jyttxC3v15C1fHt6LLKVFOh2NOwBJMgCgtr+CuGSvIKyrl\ntRHxdk2AoVerhrx4XRyrM/dx/wcrA2L48rPzUxGB+60kjF+wBBMgJsxNYenmXJ66qqcV+zOHXdjj\nFP52SXfmJu/i6bkpTodzUtZuz+O/K3dw85ntaN7ASsL4AxumHAC+Wr2TNxdvZtSAtlzZxyrrmt+6\n6Q+xbMkp4PVFm2jTuC7DT/O/u5eqKk/OXk+jurW4/RwrCeMv7AjGz6Xvyeehj1fRt01D/npJd6fD\nMT5IRHjsj905t0sMj32ezPdpWU6H9Lt9n5bFTxtzuOf8TtSvY92//sISjB/LLy5lzHtJ1A0PZdLw\neMLDbHeaYwsLDWHi9X3p3CyKO6cvJ2XXfqdDqrTyCmX8nBTaRPvn0Vcws28kP6WqPPzxarbkFDJx\nWF9OaVDH6ZCMj6tXO4zJoxOIrB3KTVOWsWd/sdMhVcpnK7aTsiufhy7qYj+i/IztLT/15uJNzFm7\ni3GDuwb9zaZM5TVvEMHbo/qxr6iUm99JpLCkzOmQjutQSZjerRpwiZWE8TuWYPzQTxuzGT8nhYt7\nnsItZ7VzOhzjZ05t2YCJw/qQvCOPe2eupNyHhy9P+TGDnXnFjBvSza7r8kOWYPzMzrwi7p6xgnZN\nIplwdW9E7ENnfr/zuzXj75f2YMG63Tw5e73T4RzT3oISJi1M5/yuTe0o3U/ZMGU/UlJWwdjpyyku\nLef1kadTr7btPlN1o86IJSOngLd/2EzbxnW5YUCs0yH9xsvukjB/GdLV6VBMFdk3lB954qt1rNi6\nj0nD+9KxqZXJMCfvb5d0Z1tuIf+YlUzrRnU5t2tTp0MCDpWEyeCa+NZ0bmZ/6/7Kusj8xGcrMnnn\n5y2MGdiei+1kp6kmoSHCi9f1oXuL+tw1YznJO/KcDgmAZ+alEhoiVhLGz1mC8QPrd+7nkU/XcHr7\naB6+qIvT4ZgAE1k7jLdH9aN+RC1unprIrjxnhy+vztzHrFU7uOXM9jb83s9ZgvFxeUWl3D4tiQYR\ntZg4rC9hobbLTPVrVr8Ok0f3I7+4lJumLuPAQWeGL6sqT81OIToynNvObu9IDKb62LeVD6uoUB78\ncCU79hUxaXg8MVG1nQ7JBLBuzevzyvC+pO7O5+4Zyykrr6jxGBamZfHzphzuOa8jUVYSxu9ZgvFh\nkxam8/X6PfzfH7sT37aR0+GYIHBOl6b847IefJeaxeNfrkO15q6RKa9Qxs9OoW3julxvJWECgo0i\n81GL0rJ4dkEaV/ZpycjT7cNmas7I09uyNaeANxdvJrZxJDedWTMX836yPJPU3fm8cn1fKwkTICzB\n+KBtuYXcM3MFXZpF8eSVPe1iSlPjHhnSjW25Rfzrq3W0ahTBhT1O8er2ikrKeW5+Gr1bN+Tint7d\nlqk5Xv2ZICKDRSRVRNJFZNxxlrtaRFREEtzTw0Vk5RGPChGJc782TETWiMhqEZkrIk2OWM/d7u0l\ni8gEb7bNW4pLyxk7fTnlFcprI+KJCA91OiQThEJChOeHxtGrZQPunbmSNZneHb485afN7NpfzKND\nutoPqgDitQQjIqHAK8AQoDswTET+54YlIhIF3AMsOTRPVaerapyqxgEjgQxVXSkiYcCLwLmq2gtY\nDdzlXs+5wOVAL1XtAfzHW23zpn/MSmbN9jyevzaO2CaRTodjglhEeChvjkogOjKcm95ZxvZ9RV7Z\nTm5BCa9+t5FB3ZpyWnsrCRNIvHkE0x9IV9VNqloCzMSVAI72L2AC4Gnw/TDgffdzcT8ixfUzpz6w\nw/3aHcB4VT0IoKp7qqUVNWjm0q3MXLaNu87tyKDuzZwOxxiaRtVhyo39KC4p5+apy8gvLq32bUz8\ndgMFJWX8ZbCVhAk03kwwLYFtR0xnuucdJiJ9gNaq+uVx1jMUd4JR1VJciWQNrsTSHXjbvVxn4CwR\nWSIi34tIv2ppRQ1ZnbmPx2Ylc1anJnb1svEpnZtF8eqIeNL3HODOGSuqdfjy1pxCpv2yhaH9WtPJ\nSsIEHG8mmGN1pB4e8ygiIcDzwIMeVyByGlCoqmvd07VwJZg+QAtcXWSPuBcPAxoBpwMPAR/KMTpz\nRWSMiCSKSGJWlm/cOja3oIQ7pi0npl5tXryuD6FWltz4mDM7NeHfV5zKorQsHpuVXG3Dl5+Zn0pY\nSAj3DbIfVYHImwkmE2h9xHQrfu3OAogCTgUWikgGrsQw69CJfrfr+LV7DCAOQFU3qusv/EPgjCO2\n96m6LAUqgCYcRVXfUNUEVU2IiYk5mfZVi/IK5d6ZK8jKP8irI/oSHRnudEjGHNN1/dtwxzkdmLFk\nK28t3nzS61u1bR9frNrBLWe1o1l9KwkTiLyZYJYBnUSknYiE40oWsw69qKp5qtpEVWNVNRb4BbhM\nVRPh8BHONbjO3RyyHeguIocywwXAoZtZ/Bc4z/3ezkA4kO2txlWXF75OY/GGbB6/vAe9WjV0Ohxj\njuuhC7twSc/mPDlnPXPX7qzyelSVJ2evp3FkOGMGWkmYQOW1BKOqZbhGeM3DlQQ+VNVkEXlcRC6r\nxCoGApmquumIde4A/gksEpHVuI5onnS/PBloLyJrcSWlUVqTlyFXwdfrdjPx23SGJrTmuv5tnA7H\nmBMKCRGevbY3ca0bct8HK1m5bV+V1vNd6h6WbM7l3kGdrCRMABMf/w72qoSEBE1MTHRk2xnZBVz6\n8g/ENo7ko9sHUKeWXe9i/Ef2gYNcOelHikrK+WzsH2gdXbfS7y2vUIa8uIjScmX+/QOpZQVc/Y6I\nJKlqwomWsz3rgKKScm6flkRoiDBpeF9LLsbvNKlXmymj+1FSVsFNU5eRV1T54cufJGWStvsAD13U\nxZJLgLO9W8NUlUc/W0Pq7nxevK7P7/rlZ4wv6dg0itdGxrM5u4A7py+ntBLDl4tKynl2QSpxrRsy\n5FQrCRPoLMHUsPd+2cJnK7bzwKDOnN3Z+VFsxpyMMzo04amrevJDejZ/+2ztCYcvT/5xM7v3H+TR\ni7tZSZggYMUua1DSllwe/2Id53dtyp3ndnQ6HGOqxTUJrdmaW8jEb9Np26QuY8859t92zoGDvLpw\nIxd0b0b/dtE1HKVxgiWYGpKVf5Cx05fTslEEzw2NI8QupjQB5IELOrMlp5AJc1NpE12XP/Zq8T/L\nTPw2naLScisJE0Ssi6wGlJVXcNeM5eQVlfLq8HgaRNiwTBNYRIQJV/cioW0jHvhwFUlb9v7m9Yzs\nAqb9soVrE1rTsWk9h6I0Nc0STA2YMC+VJZtzeeqqnnRvUd/pcIzxijq1QnnjhgSaN6jDre8msjWn\n8PBrz8xPpVZoCPcP6uRghKamWYLxstlrdvLGok3cMKAtV/Zp5XQ4xnhVdGQ4U0b3o0KV0VOXkldY\nyoqte/lq9U5uHdieplYSJqjYORgvSt+Tz0MfraJPm4b87ZL/uRWOMQGpfUw93hiZwIi3lnDbtEQq\nKqBJPSsJE4zsCMZLDhws47b3kogID2XScLvHuAku/dtFM+HqXvyyKZelGbnce34n6tW237PBxva4\nF6gqD3+8ioycQqbdfBrNG0Q4HZIxNe6KPi3JPnCQnzbmWK29IGUJxgveWryZ2Wt28ejFXRnQwW4B\na4LXLWe155azrGssWFm/TTX7eWMO4+emMOTUU7jVPljGmCBmCaYa7cor5u73lxPbuC7PXNPbSmEY\nY4KadZFVk5KyCsZOT6KopJyZY063E5rGmKBn34LV5Imv1rF86z5eub4vHZtGOR2OMcY4zrrIqsFn\nKzJ55+ct3HpWOy7p1dzpcIwxxidYgjlJ63fu55FP13Bau2gr4meMMUewBHMS8opKuWNaEg0iavHy\n9X0Js7vzGWPMYXYOpooqKpQHP1xJ5t4iPrjtdGKiajsdkjHG+BT7yV1Fr36/ka/X7+H//tid+LZ2\n8yRjjDmaJZgqWJSWxX/mp3JFXAtuGNDW6XCMMcYnWYKpgoKDZfRp3ZAnr+ppF1MaY4wHdg6mCob0\nbM5FPU6x2x4bY8xx2BFMFVlyMcaY47MEY4wxxisswRhjjPEKSzDGGGO8whKMMcYYr7AEY4wxxiss\nwRhjjPEKSzDGGGO8QlTV6RgcIyJZwJYqvr0JkF2N4TjJ2uJ7AqUdYG3xVSfTlraqGnOihYI6wZwM\nEUlU1QSn46gO1hbfEyjtAGuLr6qJtlgXmTHGGK+wBGOMMcYrLMFU3RtOB1CNrC2+J1DaAdYWX+X1\nttg5GGOMMV5hRzDGGGO8whJMJYlIhoisEZGVIpLonhctIgtEZIP730ZOx3kiHtrxDxHZ7p63UkQu\ndjrOyhCRhiLysYikiMh6ERngj/sEPLbF7/aLiHQ5It6VIrJfRO7zt/1ynHb43T4BEJH7RSRZRNaK\nyPsiUkdE2onIEvc++UBEwqt9u9ZFVjkikgEkqGr2EfMmALmqOl5ExgGNVPUvTsVYGR7a8Q/ggKr+\nx6m4qkJE3gEWq+pb7g9HXeBR/GyfgMe23Icf7pdDRCQU2A6cBtyJH+4X+J923Iif7RMRaQn8AHRX\n1SIR+RCYDVwMfKqqM0XkNWCVqr5andu2I5iTcznwjvv5O8AVDsYSVESkPjAQeBtAVUtUdR9+uE+O\n0xZ/dz6wUVW34If75QhHtsNfhQERIhKG68fLTuA84GP3617ZJ5ZgKk+B+SKSJCJj3POaqepOAPe/\nTR2LrvKO1Q6Au0RktYhM9vXuC7f2QBYwRURWiMhbIhKJf+4TT20B/9svR7oOeN/93B/3yyFHtgP8\nbJ+o6nbgP8BWXIklD0gC9qlqmXuxTKBldW/bEkzl/UFV+wJDgDtFZKDTAVXRsdrxKtABiMP1B/is\ng/FVVhjQF3hVVfsABcA4Z0OqMk9t8cf9AoC7m+8y4COnYzkZx2iH3+0TdxK8HGgHtAAicX3+j1bt\n50sswVSSqu5w/7sH+AzoD+wWkeYA7n/3OBdh5RyrHaq6W1XLVbUCeBNX23xdJpCpqkvc0x/j+pL2\nu32Ch7b46X45ZAiwXFV3u6f9cb/AUe3w030yCNisqlmqWgp8CpwBNHR3mQG0AnZU94YtwVSCiESK\nSNSh58CFwFpgFjDKvdgo4HNnIqwcT+049MF3uxJX23yaqu4CtolIF/es84F1+Nk+Ac9t8cf9coRh\n/LZbye/2i9tv2uGn+2QrcLqI1BUR4dfPynfA1e5lvLJPbBRZJYhIe1y/9sHVnTFDVZ8QkcbAh0Ab\nXDvxGlXNdSjMEzpOO97DdcivQAZw26H+cl8mInHAW0A4sAnXCJ8Q/GifHOKhLS/hn/ulLrANaK+q\nee55fvVZAY/t8NfPyj+BoUAZsAK4Bdc5l5lAtHveCFU9WK3btQRjjDHGG6yLzBhjjFdYgjHGGOMV\nlmCMMcZ4hSUYY4wxXmEJxhhjjFdYgjHGGOMVlmCM8SEiMlVErj7xksb4PkswxhhjvMISjDEnICKx\n7puAvem+adN8EYkQkYUikuBepon7XjuIyGgR+a+IfCEim0XkLhF5wF0p+RcRia7kduNF5Ht35et5\nR9TyulVElonIKhH5xF0CpIG4biYX4l6mrohsE5FaItJBROa617NYRLq6l7nGfQOqVSKyyCv/eSao\nWYIxpnI6Aa+oag9gH/CnEyx/KnA9rmKITwCF7krJPwM3nGhjIlILmAhcrarxwGT3esB1k6h+qtob\nWA/c7C5lsgo4273MpcA8d3HDN4C73ev5MzDJvcxjwEXu9Vx2opiM+b3CTryIMQZXNdqV7udJQOwJ\nlv9OVfOBfBHJA75wz18D9KrE9rrgSlILXPUJCcVVHh7gVBH5N9AQqAfMc8//AFe9qe9w3cNkkojU\nw1U59yP3egBqu//9EZjqvsPhp5WIyZjfxRKMMZVzZBHAciACV+HAQ70AdY6zfMUR0xVU7nMnQLKq\nDjjGa1OBK1R1lYiMBs5xz58FPOXugosHvsV17499qhp39EpU9XYROQ24BFgpInGqmlOJ2IypFOsi\nM6bqMnB9kcOvZc+rSyoQIyIDwNVlJiI93K9FATvd3WjDD71BVQ8AS4EXgS/d9y3ZD2wWkWvc6xER\n6e1+3kFVl6jqY0A20Lqa22CCnCUYY6ruP8AdIvIT0KQ6V6yqJbiS1tMisgpYiaurC+D/gCXAAiDl\nqLd+AIxw/3vIcOBm93qScd3dEOAZEVkjImuBRbjO4RhTbaxcvzHGGK+wIxhjjDFeYSf5jXGAiLwC\n/OGo2S+q6hQn4jHGG6yLzBhjjFdYF5kxxhivsARjjDHGKyzBGGOM8QpLMMYYY7zCEowxxhiv+H8U\n08xu7+5RNgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x28296962f60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "n_leafs = len(num_leaves_s)\n",
    "x_axis = num_leaves_s\n",
    "plt.plot(x_axis, -test_means)\n",
    "plt.xlabel( 'num_leaves' )\n",
    "plt.ylabel( 'Log Loss' )\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "虽然是50为最佳点，但由于算力不足，范围不再调整重新训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done   8 out of  12 | elapsed: 13.1min remaining:  6.5min\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  12 | elapsed: 18.3min finished\n"
     ]
    },
    {
     "ename": "AttributeError",
     "evalue": "'GridSearchCV' object has no attribute 'best_estimator_'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-31-6301de06982a>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m     20\u001b[0m \u001b[0mgrid_search\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mGridSearchCV\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlg\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mn_jobs\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m4\u001b[0m\u001b[1;33m,\u001b[0m  \u001b[0mparam_grid\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mtuned_parameters\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcv\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mkfold\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mscoring\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m\"neg_log_loss\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mverbose\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m5\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mrefit\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mFalse\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     21\u001b[0m \u001b[0mgrid_search\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX_train\u001b[0m \u001b[1;33m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 22\u001b[1;33m \u001b[0mgrid_search\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mbest_estimator_\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m: 'GridSearchCV' object has no attribute 'best_estimator_'"
     ]
    }
   ],
   "source": [
    "#调整min_child_samples\n",
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'max_bin': 127, \n",
    "          'subsample': 0.7,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "min_child_samples_s = range(10,50,10) \n",
    "tuned_parameters = dict( min_child_samples = min_child_samples_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)\n",
    "#grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.47558815927421955\n",
      "{'min_child_samples': 40}\n"
     ]
    }
   ],
   "source": [
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl4VeW5/vHvk4kpYQ5hCiRRRkGG\nREQFi2hbtC12wCpS1BbEY2uHU+2p9vRXW+1oe0ona8tghWrF8VSqrZ5WQAYVCQgyBjBMYUqYAwEy\nPb8/9iZGGpINhKy9k/tzXbnIWvvdaz+vC7n3Wu9a7zJ3R0REJC7oAkREJDooEEREBFAgiIhImAJB\nREQABYKIiIQpEEREBFAgiIhImAJBREQABYKIiIQlBF3A2ejYsaNnZGQEXYaISExZvnz5PndPratd\nTAVCRkYGubm5QZchIhJTzGxbJO10ykhERAAFgoiIhCkQREQEUCCIiEiYAkFERAAFgoiIhCkQREQE\naCKB8Myy7czfUBh0GSIiUa3RB0JZRSWz39rGV59+l017i4MuR0QkajX6QEiMj2P6bTk0T4xn0qxc\nDh4rDbokEZGo1OgDAaBr2xZMuy2bPUdOcPdTyymrqAy6JBGRqNMkAgFgaI92/PSzA3k7/wA/+Nva\noMsREYk6MTW53fn67NDu5O0t5o9v5NMnLYWJV2QEXZKISNSI6AjBzMaYWZ6ZbTaz+2tpN87M3Mxy\nwsuJZjbLzFab2XozeyC8vrmZvWNmq8xsrZn9oH66U7f/+nhfru3bie//bR1LNu9rqI8VEYl6dQaC\nmcUDjwLXA/2B8WbWv4Z2KcDXgKXVVt8ENHP3gUA2cJeZZQAngdHuPggYDIwxs+Hn15XIxMcZv7pl\nMBeltuLLT61gy75jDfGxIiJRL5IjhGHAZnfPd/dSYA5wYw3tHgYeAU5UW+dAKzNLAFoApcARDzka\nbpMY/vFz7MNZS2meyIzbLiPOYNKsZRw+XtZQHy0iErUiCYRuwI5qywXhdVXMbAiQ7u4vn/be54Fj\nwG5gO/ALdz8Qfk+8ma0ECoF/uvtSGlCPDi157AvZbN9fwleffpdyXXkkIk1cJIFgNayr+jZvZnHA\nVODeGtoNAyqArkAmcK+ZZQG4e4W7Dwa6A8PMbECNH242xcxyzSy3qKgognIjNzyrAw9/egALNxbx\nk39sqNdti4jEmkgCoQBIr7bcHdhVbTkFGAAsMLOtwHBgbnhg+VbgVXcvc/dCYAmQU33j7n4IWACM\nqenD3X2au+e4e05qap2PBD1r44f14I4rM5i5eAvPLNte79sXEYkVkQTCMqCXmWWaWRJwCzD31Ivu\nftjdO7p7hrtnAG8DY909l9BpotEW0opQWGwws1QzawtgZi2A64DAvqJ/9xP9GNmrI9/96xre2XIg\nqDJERAJVZyC4ezlwD/AasB541t3XmtlDZja2jrc/CiQDawgFy5/c/T2gCzDfzN4Lr/9nDeMPDSYh\nPo7fjR9KeruW/MeTy9lxoCSoUkREAmPuDXZxz3nLycnx3NzcC7b9/KKjfPrRJXRt24Ln776S5GZN\n6r49EWmkzGy5u+fU1a7JTF0RiazUZB6dMJRNhUf5z2dWUlkZO2EpInK+FAinGdkrle9+oh//XLeX\n//lnXtDliIg0GJ0TqcEdV2awcW8xj85/n95pKdw4uFvdbxIRiXE6QqiBmfGDsQMYltmebz3/Hit3\nHAq6JBGRC06BcAZJCXH84QvZdEppxp2zc9l9+HjQJYmIXFAKhFq0b5XEzNsvo+RkOVNmL+d4aUXQ\nJYmIXDAKhDr06ZzCr28Zwppdh7nv+VXE0mW6IiJnQ4EQgev6p/HtMX155b3d/Hbe5qDLERG5IHSV\nUYTuujqLjXuK+eU/N9KrUzLXD+wSdEkiIvVKRwgRMjN+/NmBDOnRlm8+u4o1Ow8HXZKISL1SIJyF\n5onx/HFiNm1bJjJldi5FxSeDLklEpN4oEM5Sp5TmTL8th4MlZdz151xOluvKIxFpHBQI52BAtzb8\nz+cHsWL7IR54cbWuPBKRRkGBcI5uGNiFb1zXixdX7GT6ovygyxEROW+6yug8fG10LzbtPcpP/rGB\nizslM7pvWtAliYicMx0hnIe4OOMXNw3ikq6t+drTK9m4tzjokkREzpkC4Ty1SIpn+m05tEiKZ/Ks\nXA4cKw26JBGRcxJRIJjZGDPLM7PNZnZ/Le3GmZmbWU54OdHMZpnZajNbb2YPhNenm9n88Lq1Zvb1\n+ulOMLq0acG0idnsOXKCu59cTml5ZdAliYictToDwcziCT0b+XqgPzDezPrX0C4F+BqwtNrqm4Bm\n7j4QyAbuMrMMoBy41937AcOBr9S0zVgypEc7fva5gSzdcoAH567VlUciEnMiOUIYBmx293x3LwXm\nADfW0O5h4BHgRLV1DrQyswSgBVAKHHH33e6+AsDdi4H1QMw/heYzQ7pz96iLePqd7cx+a1vQ5YiI\nnJVIAqEbsKPacgGn/eNtZkOAdHd/+bT3Pg8cA3YD24FfuPuB096bAQzhw0cW1V+fYma5ZpZbVFQU\nQbnB+tbH+nBdv0489PI6Fm2K/npFRE6JJBCshnVV50PMLA6YCtxbQ7thQAXQFcgE7jWzrGrvTQZe\nAL7h7kdq+nB3n+buOe6ek5qaGkG5wYqLM351yxAuTk3mK0+tIL/oaNAliYhEJJJAKADSqy13B3ZV\nW04BBgALzGwroTGBueGB5VuBV929zN0LgSVA1YAzoTB4yt1fPN+ORJPkZgnMuD2HhPg4Js/K5fDx\nsqBLEhGpUySBsAzoZWaZZpYE3ALMPfWiux92947unuHuGcDbwFh3zyV0mmi0hbQiFBYbzMyAmcB6\nd/9lPfcpKqS3b8ljE4ay42AJX336XcordOWRiES3OgPB3cuBe4DXCA3+Puvua83sITMbW8fbHwWS\ngTWEguVP7v4ecBUwkVBYrAz/3HA+HYlGl2d14OEbB7BwYxE//vuGoMsREalVRFNXuPvfgb+ftu57\nZ2g7qtrvRwldenp6m8XUPDbR6NwyrAd5e4t5fMkWeqclc8uwHkGXJCJSI92p3AD++4Z+jOzVkf/3\n0hqW5u8PuhwRkRopEBpAQnwcv7t1KOntWnL3UyvYcaAk6JJERP6NAqGBtGmRyIzbcyivqGTyrFyO\nniwPuiQRkQ9RIDSgrNRkHp0wlM1FR/nGnHepqNT0FiISPRQIDWxkr1S+98n+/Gt9Ib/4v7ygyxER\nqaIH5ATgtit6kre3mMcWvE/vtGQ+M6R70CWJiOgIIQhmxg/GXsLwrPZ8+4XVrNh+MOiSREQUCEFJ\njI/jsQnZdG7dnCmzl7P78PGgSxKRJk6BEKB2rZKYcXsOJ8oquHN2LsdLK4IuSUSaMAVCwHqnpfCb\n8YNZu+sI9z23Sg/WEZHAKBCiwOi+adw/pi+vrN7Nb17fHHQ5ItJE6SqjKDHl6izy9hYz9V8b6ZWW\nzA0DuwRdkog0MTpCiBJmxo8/M5ChPdryzWdXsmbn4aBLEpEmRoEQRZonxvOHidm0b5nEnbNzKSw+\nUfebRETqiQIhynRKac6023I4VFLGXX9ezokyXXkkIg1DgRCFBnRrwy8/P4h3tx/iOy+u1pVHItIg\nIgoEMxtjZnlmttnM7q+l3Tgz8/DzlDGzRDObZWarzWy9mT1Qre3jZlZoZmvOvxuNz/UDu/DNj/bm\nxXd38seF+UGXIyJNQJ2BYGbxhB6FeT3QHxhvZv1raJcCfA1YWm31TUAzdx8IZAN3mVlG+LUngDHn\nUXuj99XRF/PJS7vws1c38K91e4MuR0QauUiOEIYBm909391LgTnAjTW0exh4BKg+EupAKzNLAFoA\npcARAHdfCBw4j9obPTPj5+MGMaBrG74+513y9hQHXZKINGKRBEI3YEe15YLwuipmNgRId/eXT3vv\n88AxYDewHfiFuysEzkKLpHim35ZDq2YJTJ69jAPHSoMuSUQaqUgCwWpYVzXKaWZxwFTg3hraDQMq\ngK5AJnCvmWWdTYFmNsXMcs0st6io6Gze2mh0bhO68mjvkZPc/eRySssrgy5JRBqhSAKhAEivttwd\n2FVtOQUYACwws63AcGBueGD5VuBVdy9z90JgCZBzNgW6+zR3z3H3nNTU1LN5a6MyOL0tPx93KUu3\nHODBuWt05ZGI1LtIAmEZ0MvMMs0sCbgFmHvqRXc/7O4d3T3D3TOAt4Gx7p5L6DTRaAtpRSgsNtR7\nL5qIGwd348ujLuLpd3Yw682tQZcjIo1MnYHg7uXAPcBrwHrgWXdfa2YPmdnYOt7+KJAMrCEULH9y\n9/cAzOxp4C2gj5kVmNmk8+hHk3Hfx/rw0f5pPPTyOhZubJqn0ETkwrBYOvWQk5Pjubm5QZcRuKMn\nyxn32JvsPHScv37lKi5KTQ66JBGJYma23N3rPF2vO5VjUHKzBKbflkNifBx3zsrlcElZ0CWJSCOg\nQIhR6e1b8ocvZLPjYAn3PL2C8gpdeSQi50eBEMOGZbbnh58ewKJN+/jhK+uDLkdEYpwekBPjbr6s\nBxv3HmXm4i30Tkvh1st7BF2SiMQoHSE0Ag9c35eP9E7ley+t4e38/UGXIyIxSoHQCCTEx/HbW4fQ\ns0NL7n5yOdv3lwRdkojEIAVCI9G6eSIzbr+MSofJs5dRfEJXHonI2VEgNCKZHVvx+wlDeb/oGN+Y\ns5KKyti5x0REgqdAaGSuurgjD36qP69vKOTnr+UFXY6IxBBdZdQITRzek7w9xfzhjffpnZbMZ4d2\nD7okEYkBOkJohMyM74+9hOFZ7bn/hdWs2H4w6JJEJAYoEBqpxPg4HpuQTec2zZkyezm7Dh0PuiQR\niXIKhEasXaskZt6ew4myCu6cnUtJaXnQJYlIFFMgNHK90lL47fghrNt9hHufXUWlrjwSkTNQIDQB\n1/TtxHeu78c/1uzh169vCrocEYlSusqoiZg8MpO8vcX8+vVN9EpL5pOXdg26JBGJMjpCaCLMjB99\nZgDZPdtx33OrWLPzcNAliUiUiSgQzGyMmeWZ2WYzu7+WduPMzM0sJ7ycaGazzGy1ma03swfOdptS\nf5olxPOHL2TToVUz7pydS+GRE0GXJCJRpM5AMLN4Qs9Gvh7oD4w3s/41tEsBvgYsrbb6JqCZuw8E\nsoG7zCwj0m1K/UtNaca027I5VFLGlD8v50RZRdAliUiUiOQIYRiw2d3z3b0UmAPcWEO7h4FHgOpf\nOx1oZWYJQAugFDhyFtuUC+CSrm2YevMgVu44xAMvriaWnqstIhdOJIHQDdhRbbkgvK6KmQ0B0t39\n5dPe+zxwDNgNbAd+4e4HItlmtW1PMbNcM8stKiqKoFyJxJgBXbj3o73533d38oc38oMuR0SiQCSB\nYDWsq/pKaWZxwFTg3hraDQMqgK5AJnCvmWXVtc0PrXSf5u457p6TmpoaQbkSqXtGX8ynBnXlkdc2\n8M91e4MuR0QCFkkgFADp1Za7A7uqLacAA4AFZrYVGA7MDQ8s3wq86u5l7l4ILAFyItimNAAz4+fj\nLmVgtzZ8Y867bNhzJOiSRCRAkQTCMqCXmWWaWRJwCzD31IvuftjdO7p7hrtnAG8DY909l9BpotEW\n0opQWGyoa5vScJonxjNtYg6tmiUweVYu+4+eDLokEQlInYHg7uXAPcBrwHrgWXdfa2YPmdnYOt7+\nKJAMrCEUAn9y9/fOtM3z6Iech85tmjPtthyKik9y95MrKC2vDLokEQmAxdIVJjk5OZ6bmxt0GY3W\nSyt38vU5K7k5J52ffm4gZjUN9YhIrDGz5e6eU1c7TV0hVW4c3I1Ne4/yu/mb6dM5hS+NyAy6JBFp\nQJq6Qj7kmx/tzcf6p/HDV9bxxkZd5ivSlCgQ5EPi4oypNw+md1oK9/xlBe8XHQ26JBFpIAoE+Tet\nmiUw4/YckuLjmDwrl8MlZUGXJCINQIEgNereriV/nJhNwcESvvKXFZRX6MojkcZOgSBnlJPRnh99\neiCLN+/jh6+sD7ocEbnAdJWR1Orzl6WzcW8xMxZvoVdaMhMu7xl0SSJygegIQer0wA39+EjvVB58\naS1vvb8/6HJE5AJRIEid4uOM3946hJ4dWnL3U8vZvr8k6JJE5AJQIEhEWjdPZObtl+EOk2Yto/iE\nrjwSaWwUCBKxjI6teGzCUPL3HePrc1ZSURk7056ISN0UCHJWrry4I98fewnzNhTyyKsbgi5HROqR\nrjKSszZxeE827inmjwvz6ZWWwrjs7kGXJCL1QEcIck6+96n+XHlRB77z4mqWbzsQdDkiUg8UCHJO\nEuPj+P2EoXRp25y7/rycnYeOB12SiJwnBYKcs7Ytk5h5ew4nyyq5c1YuJaXlQZckIuchokAwszFm\nlmdmm83s/lrajTMzDz9PGTObYGYrq/1Umtng8Gs3m9l7ZrbWzB6pn+5IQ7u4Uwq/uXUIG/Yc4d5n\nV1GpK49EYladgWBm8YQehXk90B8Yb2b9a2iXAnwNWHpqnbs/5e6D3X0wMBHY6u4rzawD8HPgWne/\nBEgzs2vrpUfS4K7p04nv3NCPf6zZw69e3xR0OSJyjiI5QhgGbHb3fHcvBeYAN9bQ7mHgEeDEGbYz\nHng6/HsWsNHdTz2B5V/A5yKuWqLOpBGZ3JTdnd+8vom/rdoVdDkicg4iCYRuwI5qywXhdVXMbAiQ\n7u4v17Kdm/kgEDYDfc0sw8wSgE8D6RFXLVHHzPjhZwaQ07Md9z23ivcKDgVdkoicpUgCoaYnrVed\nKDazOGAqcO8ZN2B2OVDi7msA3P0gcDfwDLAI2ArUOCJpZlPMLNfMcouK9EjHaNYsIZ4/TMymY3Iz\n7pydy94jZzpYFJFoFEkgFPDhb+/dgernBFKAAcACM9sKDAfmnhpYDruFD44OAHD3v7n75e5+BZAH\n1Hjy2d2nuXuOu+ekpqZGUK4EqWNyM6bflkPxiXKmzM7lRFlF0CWJSIQiCYRlQC8zyzSzJEL/uM89\n9aK7H3b3ju6e4e4ZwNvAWHfPhaojiJsIjT1UMbNO4T/bAV8GZtRDfyQK9O/amqk3D2ZVwWG+/cJ7\nuOvKI5FYUGcguHs5cA/wGrAeeNbd15rZQ2Y2NoLPuBoocPf809b/2szWAUuAn7r7xrOsXaLYxy/p\nzLc+3oeXVu7i9wveD7ocEYmAxdK3t5ycHM/NzQ26DImQu/P1OSuZu2oX0yZm87FLOgddkkiTZGbL\n3T2nrna6U1kuGDPjkXGXMqh7G77xzErW7z4SdEkiUgsFglxQzRPjmXZbDinNE5g8K5f9R08GXZKI\nnIECQS64tNbNmX5bDvuOnuQ/nlxOaXll0CWJSA0UCNIgLu3elp/fNIhlWw/y3b+u1pVHIlFID8iR\nBjN2UFc27S3mt/M206dzayaNyAy6JBGpRkcI0qD+87refPySNH70yjoW5BUGXY6IVKNAkAYVF2f8\n8vOD6dO5NV95agW/eC2PwmJNcSESDRQI0uBaNUvg8TtyuPLijjy6YDMjfjqf+55bxYY9uixVJEi6\nMU0CtWXfMR5fvIXnlu/gRFklI3t1ZPLILK7u1RGzmuZVFJGzFemNaQoEiQoHj5Xyl3e288SbWykq\nPknvtGQmj8jixiFdaZYQH3R5IjFNgSAx6WR5BX9btZsZi/LZsKeYjsnNuO2KnnxheE/at0oKujyR\nmKRAkJjm7izZvJ/pi/J5Y2MRzRLi+Fx2dyaNyOSi1OSgyxOJKZEGgu5DkKhkZozo1ZERvTqycW8x\nMxdt4fncAv6ydDvX9evEpBFZDM9qr3EGkXqkIwSJGUXFJ/nz29t48u1tHDhWyoBurZk8IotPXNqF\nxHhdMCdyJjplJI3WibIKXlyxkxmL88kvOkbn1s2546oMxg/rQZsWiUGXJxJ1FAjS6FVWOgs2FjJ9\n4Rbeyt9Pq6R4Pn9ZOl+6KpP09i2DLk8kaigQpElZs/MwMxdv4W+rdlHpzpgBnZk0Iovsnu2CLk0k\ncPX6gBwzG2NmeWa22czur6XdODNzM8sJL08ws5XVfirNbHD4tfFmttrM3jOzV82sY6SdEzndgG5t\nmHrzYBZ9+xqmXH0Rizft43OPvclnf7+Ef6zeTUVl7HzxEQlKnUcIZhYPbAQ+ChQAy4Dx7r7utHYp\nwCtAEnCPu+ee9vpA4CV3zzKzBGAX0N/d95nZI0CJu3+/tlp0hCCROnaynOdydzBzyRZ2HDhOevsW\nfOmqTD6fk06rZrq4TpqW+jxCGAZsdvd8dy8F5gA31tDuYeAR4EwzlY0Hnj5VX/inlYWuG2xNKCBE\n6kWrZgnccVUmC+67hscmDKVTSnN+8Ld1DP/J6/zkH+vZffh40CWKRJ1IAqEbsKPackF4XRUzGwKk\nu/vLtWznZsKB4O5lwN3AasJHCsDMmt5kZlPMLNfMcouKiiIoV+QD8XHG9QO78MLdV/Lil6/k6l6p\nTF+Yz8ifzecbc95lzc7DQZcoEjUiOXau6c6fqvNMZhYHTAXuOOMGzC4ndEpoTXg5kVAgDAHygd8C\nDwA//LcPcp8GTIPQKaMI6hWp0dAe7Rg6oR07DpTwpyVbeWbZdv66chfDs9pz58gsrunTibg43egm\nTVckRwgFQHq15e58+PROCjAAWGBmW4HhwNxTA8tht/DB6SKAwQDu/r6HBjGeBa486+pFzkF6+5Z8\n71P9efOBa3ng+r5s21/CpFm5XDf1DZ5auo0TZRVBlygSiEgGlRMIDSpfC+wkNKh8q7uvPUP7BcB9\npwaVw0cQ24Gr3T0/vK4rsBy41N2LzOxhoKW731tbLRpUlguhrKKSv6/ezfRF+azZeYT2rZL4wuU9\nmHhFBqkpzYIuT+S81dtcRu5ebmb3AK8B8cDj7r7WzB4Cct19bh2buBooOBUG4W3uMrMfAAvNrAzY\nRi2nnEQupMT4OG4c3I2xg7qydMsBZizawm/nb+YPb+Tz6SFdmTwyi95pKUGXKXLB6cY0kRrkFx1l\n5uItvLCigBNllXykdyqTR2Yy4mI9uEdij+5UFqkHB46V8tTb25j11jb2HT1J384pTBqRydjBenCP\nxA4Fgkg9OllewUsrdzFz0Rby9haTmtKM26/oyYTLe9JOD+6RKKdAELkA3J1Fm/YxfVE+izbto3li\nHDdlp/OlEZlkdmwVdHkiNVIgiFxgeXuKmbEon5dW7qKsspLr+qUxeUQmwzL14B6JLgoEkQZSWHyC\nP78VenDPwZIyLu3ehkkjMrlhoB7cI9FBgSDSwI6XVvDCigIeX7yF/H3H6NqmOV+8KpObh6XTurke\n3CPBUSCIBKSy0pm3oZDpi/JZuuUAyc0SuPmydO64MkMP7pFAKBBEosDqgsPMWJzPy+/txt25fmAX\n7hyZxeD0tkGXJk2IAkEkiuw6dJxZb27lL+9sp/hEOTk92zF5ZBYf7Z9GvCbUkwtMgSAShY6eLOfZ\nZTt4fMkWCg4ep2eHlnzpqkzGZXfXg3vkglEgiESx8opK/m/dXqYvyufd7Ydo0yKRWy/vwR1XZpDW\nunnQ5Ukjo0AQiRHLt4Um1Htt7R7i44xPXdqVSSMzuaRrm6BLk0ai3mY7FZELK7tne7J7tmf7/hIe\nX7KFZ3N38OK7O7nyog7cOTKLj/RO1YN7pEHoCEEkyhwuKePpZdt5YslW9hw5wcWdkpk0IpPPDOlG\n80RNqCdnT6eMRGJcaXklr6zexfSFW1i3+wgdWiXxheE9mXhFTzom68E9EjkFgkgj4e68lb+fmYu2\n8PqGQpIS4vjc0G5MGpHJxZ304B6pW6SBENFEK2Y2xszyzGyzmd1fS7txZuannqdsZhPMbGW1n0oz\nG2xmKaet32dmv4q8eyJNh5lx5UUdmXnHZfzrmx9hXHZ3Xlyxk+t+uZAv/ukdlmzeRyx9sZPoFckz\nleMJPVP5o0ABoWcqj3f3dae1SwFeAZKAe049U7na6wOBl9w9q4bPWA78p7svrK0WHSGIhOw/epIn\n397On9/eyr6jpfTv0prJIzP55KVdSUrQhHryYfV5hDAM2Ozu+e5eCswBbqyh3cPAI8CJM2xnPPB0\nDYX2AjoBiyKoRUSADsnN+Pp1vVj87dH87HMDKauo5JvPrmLkI/P4/YLNHCopDbpEiUGRBEI3YEe1\n5YLwuipmNgRId/eXa9nOzdQQCISC4hnXMa/IWWueGM/Nl/Xg//7zap744mX0TkvhkVfzuOIn83jw\npTVs238s6BIlhkRyH0JNF0BX/eNtZnHAVOCOM27A7HKgxN3X1PDyLcDEWt47BZgC0KNHjwjKFWl6\nzIxRfToxqk8n1u8+woxFW/jLO9uZ/fY2PtY/jTtHZpHds50e3CO1imQM4Qrg++7+8fDyAwDu/pPw\nchvgfeBo+C2dgQPA2FPjCGY2FShy9x+ftu1BwHPu3juSYjWGIBK5wiMnmPXWVp58ezuHj5cxKL0t\nd47MZMwlnUnQg3ualHq77NTMEggNKl8L7CQ0qHyru689Q/sFwH3VwiAO2A5c7e75p7X9KXDS3R+s\ns0coEETORUlpOS8sL2Dm4i1s3V9Ct7Yt+OJVGdx8WTopenBPk1Bvg8ruXg7cA7wGrAeedfe1ZvaQ\nmY2NoJargYLTwyDs89Q8riAi9aRlUgITr8jg9XtHMW1iNt3atuCHr6znyp/M40evrGPnoeNBlyhR\nQjemiTRBq3YcYsbiLfx99W4AbhjYhTtHZnJpdz24pzHSncoiUqedh47zxJItzHlnB8UnyxmW0Z6b\ncrozqk8nUlM0PUZjoUAQkYgVnyjjmWU7eOLNrRQcDJ1CGtS9Ddf07cTovp0Y0LWNZlyNYQoEETlr\n7s7aXUeYv6GQeXmFrNxxCHdITWnGqN6pjO7biRG9OmowOsYoEETkvO0/epI3NhYxb0MhCzcWceRE\nOYnxxmUZ7RndtxPX9O1EVsdWur8hyikQRKRelVdUsnzbQeblFTJ/QyEb94ZuPerZoSXX9AmFw+WZ\n7fXMhiikQBCRC6rgYEno1NKGQt58fz8nyytpkRjPVRd3ZHR47KFzGz0fOhooEESkwRwvreCt/H3M\n21DI/A1FVfc29OvSmtF9Q2MPg9PbEa+B6UAoEEQkEO7Oxr1HmZ8XOnpYvu0gFZVOu5aJfKR3Ktf0\n7cRHeqfStmVS0KU2GQoEEYl6NGttAAAJLUlEQVQKh0vKWLipiPkbClmwsYgDx0qJM8ju2a7qstY+\naSkamL6AFAgiEnUqKp1VBYeqxh7W7joCQNc2zavC4cqLOtIiSQPT9UmBICJRb++RE1XhsHjzPkpK\nK0hKiOOKrA5VA9Pp7VsGXWbMUyCISEw5WV7BO1sOhAemC9m6vwSAizslh+556NOJnIx2JGrq7rOm\nQBCRmJZfdJR5GwpZkFfE0i37KatwUponcHWv0MD0qD6pdEzWfEuRUCCISKNx9GQ5izftY/6GQubn\nFVJYfBIzuLR7W0b3CZ1auqRra823dAYKBBFplCornXW7jzAvPPawquCD+Zau6XNqvqVUkptF8oTg\npkGBICJNwr6jJ3kjr4h5eaH5lorD8y0Ny2zPNeGjh6zU5KDLDJQCQUSanLLwfEunrlzaVBiabymj\nQ0tGhcPh8qz2NEtoWpe11msgmNkY4NdAPDDD3X96hnbjgOeAy9w918wmAN+q1uRSYKi7rzSzJOB3\nwCigEvhvd3+htjoUCCJyNnYcKKm6Y/qt8HxLLZM+mG/pmj5NY76legsEM4sHNgIfBQqAZcB4d193\nWrsU4BUgCbjH3XNPe30g8JK7Z4WXfwDEu/t3zSwOaO/u+2qrRYEgIufqeGkFb76/r+qy1l2HTwDQ\nv0vrqqm8B6e3bZTzLUUaCJGMugwDNrt7fnjDc4AbgXWntXsYeAS47wzbGQ88XW35S0BfAHevBGoN\nAxGR89EiKZ5r+6Vxbb803J28vcXM3xCaUuOxN97nd/M3075V0gfzLfVKpU3LpvUgoEgCoRuwo9py\nAXB59QZmNgRId/eXzexMgXAzoSDBzE49yfthMxsFvE/oqGLv6W8ysynAFIAePXpEUK6ISO3MjL6d\nW9O3c2vuHnURh0vKeOPUfEt5hfzvuzuJjzOye3ww31LvtORGP99SJIFQ03+BqvNM4dM9U4E7zrgB\ns8uBEndfU+1zuwNL3P2bZvZN4BfAxH/7IPdpwDQInTKKoF4RkbPSpmUiYwd1ZeygrlRUOit3fDDf\n0s9e3cDPXt1At7YtuKZvKtf0abzzLUUSCAVAerXl7sCuasspwABgQTg9OwNzzWxstXGEW/jw6aL9\nQAnwv+Hl54BJZ129iEg9i48zsnu2I7tnO+77eB/2HD5RNTD94oqdPPn2dpolxHHFRR2qBqYby3xL\nkQwqJxAaVL4W2EloUPlWd197hvYLgPtOhUH4CGI7cPWpcYjw+jnANHefZ2Z3AJ9w95tqq0WDyiIS\npJPlFSzND8+3lFfItvB8S71OzbfUtxPZPaNvvqV6G1R293Izuwd4jdBlp4+7+1ozewjIdfe5dWzi\naqCgehiEfRv4s5n9CigCvlhXLSIiQWqWEM/VvVO5uncqD3p/8vcdqzq1NHPxFv64MD8031LvVEb3\nCc231CGG5lvSjWkiIvWg+EQZizeFL2vNK2Lf0dB8S4O6t62ayvuSrq0DGZjWncoiIgGprHTW7grP\nt5RXyHvh+ZY6pTRjVADzLSkQRESixL6jJ1mQF7qsdeHGIopPNux8SwoEEZEoVFZRSe7Wg1VXLm2u\nNt/SqXsehmXW73xLCgQRkRiw40BJ1VTeb+Xvp7S8klbV51vq24m01uc335ICQUQkxpSUlvPm5v3M\nywvNt7Q7PN/SJV1bM/tLw875iqX6nMtIREQaQMukBK7rn8Z1/UPzLW3YU8z8vEJW7ThE+1ZJF/zz\nFQgiIlHIzOjXpTX9urRusM+MrtvpREQkMAoEEREBFAgiIhKmQBAREUCBICIiYQoEEREBFAgiIhKm\nQBARESDGpq4wsyJg2zm+vSOwrx7LCVJj6Utj6QeoL9GqsfTlfPvR091T62oUU4FwPswsN5K5PGJB\nY+lLY+kHqC/RqrH0paH6oVNGIiICKBBERCSsKQXCtKALqEeNpS+NpR+gvkSrxtKXBulHkxlDEBGR\n2jWlIwQREalFowwEM3vczArNbE21de3N7J9mtin8Z7sga4zEGfrxfTPbaWYrwz83BFljpMws3czm\nm9l6M1trZl8Pr4+p/VJLP2Juv5hZczN7x8xWhfvyg/D6TDNbGt4nz5jZhX8yy3mqpS9PmNmWavtl\ncNC1RsLM4s3sXTN7ObzcIPukUQYC8AQw5rR19wOvu3sv4PXwcrR7gn/vB8BUdx8c/vl7A9d0rsqB\ne929HzAc+IqZ9Sf29suZ+gGxt19OAqPdfRAwGBhjZsOBnxHqSy/gIDApwBojdaa+AHyr2n5ZGVyJ\nZ+XrwPpqyw2yTxplILj7QuDAaatvBGaFf58FfLpBizoHZ+hHTHL33e6+Ivx7MaG/7N2Isf1SSz9i\njoccDS8mhn8cGA08H14f9fsEau1LzDGz7sAngBnhZaOB9kmjDIQzSHP33RD6nxroFHA95+MeM3sv\nfEopqk+x1MTMMoAhwFJieL+c1g+Iwf0SPjWxEigE/gm8Dxxy9/JwkwJiJPBO74u7n9ovPwrvl6lm\ndm5PqW9YvwL+C6gML3eggfZJUwqExuIx4CJCh8W7gf8JtpyzY2bJwAvAN9z9SND1nKsa+hGT+8Xd\nK9x9MNAdGAb0q6lZw1Z1bk7vi5kNAB4A+gKXAe2BbwdYYp3M7JNAobsvr766hqYXZJ80pUDYa2Zd\nAMJ/FgZczzlx973hv/iVwHRC/xPHBDNLJPSP6FPu/mJ4dcztl5r6Ecv7BcDdDwELCI2LtDWzhPBL\n3YFdQdV1Lqr1ZUz4FJ+7+0ngT0T/frkKGGtmW4E5hE4V/YoG2idNKRDmAreHf78deCnAWs7ZqX88\nwz4DrDlT22gSPg86E1jv7r+s9lJM7Zcz9SMW94uZpZpZ2/DvLYDrCI2JzAfGhZtF/T6BM/ZlQ7Uv\nG0bovHtU7xd3f8Ddu7t7BnALMM/dJ9BA+6RR3phmZk8DowjNELgXeBD4K/As0APYDtzk7lE9YHuG\nfowidFrCga3AXafOwUczMxsBLAJW88G50e8QOv8eM/ulln6MJ8b2i5ldSmiAMp7Ql8Nn3f0hM8si\n9O20PfAu8IXwN+yoVUtf5gGphE67rAT+o9rgc1Qzs1HAfe7+yYbaJ40yEERE5Ow1pVNGIiJSCwWC\niIgACgQREQlTIIiICKBAEBGRMAWCiIgACgQREQlTIIiICAD/H7CMrj7j3TeBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2829680dda0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "x_axis = min_child_samples_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "#plt.errorbar(x_axis, -test_scores, yerr=test_stds ,label = ' Test')\n",
    "#plt.errorbar(x_axis, -train_scores, yerr=train_stds,label =  +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  12 out of  15 | elapsed: 33.4min remaining:  8.3min\n",
      "[Parallel(n_jobs=4)]: Done  15 out of  15 | elapsed: 43.2min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "       error_score='raise',\n",
       "       estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt', class_weight=None,\n",
       "        colsample_bytree=0.7, importance_type='split', learning_rate=0.1,\n",
       "        max_bin=127, max_depth=7, min_child_samples=40,\n",
       "        min_child_weight=0.001, min_split_gain=0.0, n_estimators=428,\n",
       "        n_jobs=4, num_class=9, num_leaves=70, objective='multiclass',\n",
       "        random_state=None, reg_alpha=0.0, reg_lambda=0.0, silent=False,\n",
       "        subsample=1.0, subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid=True, n_jobs=4,\n",
       "       param_grid={'subsample': [0.5, 0.6, 0.7, 0.8, 0.9]},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#调整sub_samples\n",
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'min_child_samples':40,\n",
    "          'max_bin': 127,\n",
    "          #'subsample': 0.7,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "subsample_s = [i/10.0 for i in range(5,10)]\n",
    "tuned_parameters = dict( subsample = subsample_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.47558815927421955\n",
      "{'subsample': 0.7}\n"
     ]
    }
   ],
   "source": [
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAD8CAYAAABpcuN4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl4FfX1+PH3yQIJENYEZAmEHcMW\n4LLZilqr4gYqu4giKi6ltGK1+qVWKz+LYpWqxQoqi4qytUpcaV1orQISMIQdQmQJi4QtCAFCkvP7\n407iFbPckGVubs7ree6Tmc98ZubMiHPubOeKqmKMMcYUJcTtAIwxxgQ2SxTGGGOKZYnCGGNMsSxR\nGGOMKZYlCmOMMcWyRGGMMaZYliiMMcYUyxKFMcaYYlmiMMYYU6wwtwMoD9HR0RoXF+d2GMYYU6Ws\nWbPmkKrGlNQvKBJFXFwcSUlJbodhjDFViojs8qefXXoyxhhTLEsUxhhjimWJwhhjTLEsURhjjCmW\nJQpjjDHFskRhjDGmWJYojDHGFKtaJ4pvD51k2sdbsJ+DNcaYolXrRPHvTQd4afkOpn+y3e1QjDEm\nYAXFm9nn666L25B68AQvfLqd2AaRDPPEuh2SMcYEnGqdKESEJ2/syr5jp3nkn+tpVj+Sn7WLdjss\nY4wJKNX60hNAeGgIL93Sk3aN63DPG2vYeuB7t0MyxpiA4leiEJGBIrJVRFJF5OFi+g0VERURjzM+\nWkSSfT55IpLgTBshIikislFEpvkso6aILHTWtUpE4sq2iSWrGxHO7LG9iawRyri5qzl4/HRFr9IY\nY6qMEhOFiIQCM4CrgXhglIjEF9IvCpgIrMpvU9X5qpqgqgnAGGCnqiaLSCPgGeByVe0MNBGRy53Z\n7gCOqmo7YDrwdJm20E/N6kcye2xvjmZlM27eak6eyamM1RpjTMDz54yiD5Cqqmmqmg0sAAYX0m8K\nMA0o6uv4KOBtZ7gNsE1VM5zxT4AhzvBgYJ4zvAS4XETEjzjLrEvzesy4uSeb9h3n129/Q05uXmWs\n1hhjApo/iaI5sMdnPN1pKyAiPYBYVX2/mOWM4IdEkQp0EpE4EQkDbgDyHzkqWJ+q5gCZQCM/4iwX\nl3VqzBODu/DZloP86b1N9o6FMaba8+epp8K+zRccPUUkBO8lorFFLkCkL5ClqhsAVPWoiNwLLATy\ngK/wnmWUuD6fZY4HxgO0bNnSj83w3y39WrHnSBYz/5tGq0a1uPPiNiXPZIwxQcqfM4p0fvi2D9AC\n2OczHgV0AZaLyE6gH5CYf0PbMZIfziYAUNX3VLWvqvYHtgL5b70VrM8526gHHDk3KFWdpaoeVfXE\nxJT4S36l9vuBnbim6wU8+eFmPlq/v9yXb4wxVYU/iWI10F5EWotIDbwH/cT8iaqaqarRqhqnqnHA\nSmCQqiZBwRnHMLz3NgqISGPnbwPgPuBVZ1IicJszPBT4TF24/hMSIjw3PIEesfX57cJk1u4+Wtkh\nGGNMQCgxUTj3CSYAy4DNwCJV3SgiT4jIID/WMQBIV9W0c9qfF5FNwJfAU6q6zWl/DWgkIqnAJKDI\nx3ErWkR4KK/c6uGCehHcNS+JXYdPuhWKMca4RoLhZq3H49GkpKQKW35axglu+vtXNKxVg3/edxH1\na9WosHUZY0xlEZE1quopqV+1fzPbH21i6vDKrR7Sj51i/OtrOJOT63ZIxhhTaSxR+Kl3XEOeHdad\nr3ce4cHFKeTlVf0zMWOM8Ue1LgpYWtd3b0b60VM8/fEWYhtG8uBVndwOyRhjKpwlilK655I27D6S\nxYzPdxDboBYj+5TvOxzGGBNoLFGUkogwZXBn9h07xeR3N9CsfiQDOpT/exzGGBMo7B7FeQgLDWHG\n6J50aBLFffPXsnn/cbdDMsaYCmOJ4jzVqRnG7LEe6tQMY9zc1RzItNLkxpjgZImiDJrW85YmP37q\nLOPmruaElSY3xgQhSxRlFN+sLjNG92Trd98z4a21VprcGBN0LFGUg0s7NmbK4C4s35rBY4kbrTS5\nMSao2FNP5eTmvi3ZczSLvy/fQcuGtbj7krZuh2SMMeXCEkU5evDKjqQfPcXUj7bQvEEk13Vr5nZI\nxhhTZpYoylFIiPDM0G4cyDzFpEXruKBuBJ64hm6HZYwxZWL3KMpZRHgos8Z4aF4/krteT+LbQ1aa\n3BhTtVmiqAANatdgztjeiAi3z/maIyez3Q7JGGPOmyWKChIXXZtXbvWwL/M0d72exOmzVprcGFM1\nWaKoQL1aNeCvIxJYs+soDyxeZ6XJjTFVkiWKCnZN16b83zWd+CBlP9OWbXU7HGOMKTV76qkS3HWx\ntzT5y//ZQWzDSEb3beV2SMYY4ze/zihEZKCIbBWRVBF5uJh+Q0VERcTjjI8WkWSfT56IJDjTRonI\nehFJEZGPRSTaaX9cRPb6zHNNeWyom0SEx6/vzGUdY/jj0o18vvWg2yEZY4zfSkwUIhIKzACuBuKB\nUSISX0i/KGAisCq/TVXnq2qCqiYAY4CdqposImHA88BlqtoNSAEm+Cxuev58qvphGbYvYISFhvC3\nm3vS6YIoJsxfy8Z9mW6HZIwxfvHnjKIPkKqqaaqaDSwABhfSbwowDSiq3vYo4G1nWJxPbRERoC6w\nrzSBV0W1a4Yxe2xv6kaGM27uavYdO+V2SMYYUyJ/EkVzYI/PeLrTVkBEegCxqvp+McsZgZMoVPUs\ncC+wHm+CiAde8+k7wbkkNVtEGhS2MBEZLyJJIpKUkZHhx2YEhiZ1I5hze2+yzuQybu5qvj991u2Q\njDGmWP4kCimkreA5TxEJAaYDDxS5AJG+QJaqbnDGw/Emih5AM7yXnh5xuv8daAskAPuBZwtbpqrO\nUlWPqnpiYqrWT5F2uqAuf7+lF6kHT3Df/LWctdLkxpgA5k+iSAdifcZb8OPLRFFAF2C5iOwE+gGJ\n+Te0HSP54bITeJMAqrpDvTW5FwEXOW3fqWququYBr+C99BV0ft4+mj/f2JUvth/i0Xc3WGlyY0zA\n8ufx2NVAexFpDezFe9C/OX+iqmYC0fnjIrIc+J2qJjnjIcAwYIDPMvcC8SISo6oZwBXAZqd/U1Xd\n7/S7EdhwfpsW+Ib3jmXP0Sxe/CyV2Ia1+NVl7dwOyRhjfqLERKGqOSIyAVgGhAKzVXWjiDwBJKlq\nYgmLGACkq2qazzL3icifgP+KyFlgFzDWmTzNeYRWgZ3A3aXcpipl0hUd2H0ki2eWbaVFg0gGJzQv\neSZjjKlEEgyXPDwejyYlJbkdxnk7k5PLmNe+Jnn3Md68sy99WltpcmNMxRORNarqKamflfAIADXD\nQpk1phctGkYy/o0kdmSccDskY4wpYIkiQNSvVYO5Y/sQKsLtc1Zz+MQZt0MyxhjAEkVAadmoFq/e\n5uG746e500qTG2MChCWKANOjZQOeH5lA8p5j/HZBspUmN8a4zhJFABrYpSmTr7mQjzceYOpHm90O\nxxhTzVmZ8QB1x89bs+dIFq988S2xDWtxa/84t0MyxlRTligClIjwx+s7s/fYKR5P3Ejz+pFcfmET\nt8MyxlRDdukpgIWGCC+M6kHnZvWY8NY3rE+30uTGmMpniSLA1aoRxmtjPTSsXYNx81az10qTG2Mq\nmSWKKqBxlLc0+emzudw+52uOW2lyY0wlskRRRXRoEsXMW3qRlnGSe99cQ3aOlSY3xlQOSxRVyEXt\nonlqSDe+TD3M5HfWW2lyY0ylsKeeqpihvVqw50gWz3+6ndiGtZh4eXu3QzLGBDlLFFXQb3/Znj1H\nsnju39uIbRjJjT1auB2SMSaIWaKogkSEp4Z0Y3/maR5aksIFdSPp37aR22EZY4KU3aOoomqEhfDy\nLb1o1ag2d7+RROrB790OyRgTpCxRVGH1aoUzZ2xvaoSFMnbOajK+t9LkxpjyZ4miiottWIvZYz0c\nPpHNnfNWcyrbSpMbY8qXJYog0K1FfV4Y1YOUvZn8ZsE35FppcmNMOfIrUYjIQBHZKiKpIvJwMf2G\nioiKiMcZHy0iyT6fPBFJcKaNEpH1IpIiIh+LSLTT3lBE/i0i252/DcpjQ4PdFfFNeOy6eP616Tue\n/MBKkxtjyk+JiUJEQoEZwNVAPDBKROIL6RcFTARW5bep6nxVTVDVBGAMsFNVk0UkDHgeuExVuwEp\nwARntoeBT1W1PfCpM278MPZnrRn3s9bM/vJb5nz5rdvhGGOChD9nFH2AVFVNU9VsYAEwuJB+U4Bp\nwOkiljMKeNsZFudTW0QEqAvsc6YNBuY5w/OAG/yI0TgmX3shV8Y34Yn3N/GvjQfcDscYEwT8SRTN\ngT0+4+lOWwER6QHEqur7xSxnBE6iUNWzwL3AerwJIh54zenXRFX3O/32A40LW5iIjBeRJBFJysjI\n8GMzqofQEOH5kT3o1rweExd8w7o9x9wOyRhTxfmTKKSQtoK7pSISAkwHHihyASJ9gSxV3eCMh+NN\nFD2AZngvPT3if9igqrNU1aOqnpiYmNLMGvQia4Ty6m29ia5TkzvmrWbPkSy3QzLGVGH+JIp0INZn\nvAU/XCYCiAK6AMtFZCfQD0jMv6HtGMkPl50AEgBUdYd6K9stAi5ypn0nIk0BnL8H/d4aUyAmqiZz\nb+9Ndk4et89dTWaWlSY3xpwffxLFaqC9iLQWkRp4D/qJ+RNVNVNVo1U1TlXjgJXAIFVNgoIzjmF4\n723k2wvEi0j+qcAVQP6jOonAbc7wbcDS89oyQ7vGUcwc42HX4ZPc/WaSlSY3xpyXEhOFqubgfSJp\nGd6D+SJV3SgiT4jIID/WMQBIV9U0n2XuA/4E/FdEUvCeYfzZmfwUcIWIbMebQJ4qzQaZH+vfthHT\nhnZjZdoRHv5HipUmN8aUmgTDgcPj8WhSUpLbYQS0Fz/dzrP/3sZvLm/P/Vd0cDscY0wAEJE1quop\nqZ9Vj60mJvyiHbt9fsdiaC8rTW6M8Y8limpCRPjzTV3Zn3mah/+RQtN6EfysXbTbYRljqgCr9VSN\nhIeG8NItPWkTU5t73lzDtu+sNLkxpmSWKKqZuhHhzLm9DxHhodw+ZzUHjxf1Ir0xxnhZoqiGmteP\nZM7Y3hzNymbcvNWcPJPjdkjGmABmiaKa6tK8Hi+O6sGmfceZ+LaVJjfGFM0SRTV2+YVN+NOgzny6\n5SB/em+jvWNhjCmUPfVUzY3pH8fuI1m88sW3tGxYizsvbuN2SMaYAGOJwvDI1ReSfvQUT364mRYN\nIhnYpanbIRljAohdejKEhAjTRySQEFuf3yxI5pvdR90OyRgTQCxRGAAiwkN59VYPTepGcOe8JHYf\nttLkxhgvSxSmQKM63tLkuaqMnfs1x7Ky3Q7JGBMALFGYH2kTU4dZYzykHznF+DfWcCYn1+2QjDEu\ns0RhfqJP64Y8M6wbX397hIeWpJBn71gYU63ZU0+mUIMTmpN+9BTPLNtKbINa/O6qjm6HZIxxiSUK\nU6T7Lm3LniNZ/O3zVGIbRjKid0u3QzLGuMAShSmSiDDlhi7sPXaK/3tnA03rRTKgQ0zJMxpjgord\nozDFCg8N4aXRPWnfuA73zV/LlgPH3Q7JGFPJLFGYEkVFhDN7bG9q1/SWJj984ozbIRljKpFfiUJE\nBorIVhFJFZGHi+k3VERURDzO+GgRSfb55IlIgohEndN+SET+6swzVkQyfKbdWT6basqiWf1IXrut\nN4dPZDP5nQ1WQNCYaqTERCEiocAM4GogHhglIvGF9IsCJgKr8ttUdb6qJqhqAjAG2Kmqyar6fX67\nM20X8E+fxS30mf5qmbbQlJsuzesx6coOfLzxAP9cu9ftcIwxlcSfM4o+QKqqpqlqNrAAGFxIvynA\nNKCon0wbBbx9bqOItAcaA1/4FbFx1V0Xt6FPXEMeT9zI3mOn3A7HGFMJ/EkUzYE9PuPpTlsBEekB\nxKrq+8UsZwSFJAq8CWSh/vhaxhARSRGRJSISW9jCRGS8iCSJSFJGRoYfm2HKQ2iI8Ozw7uSp8sCi\nZHsZz5hqwJ9EIYW0FRwdRCQEmA48UOQCRPoCWaq6oZDJI/lxAnkPiFPVbsAnwLzClqmqs1TVo6qe\nmBh7ZLMyxTasxWPXd2Zl2hFmf/mt2+EYYyqYP4kiHfD9Vt8C2OczHgV0AZaLyE6gH5CYf0PbcW4y\nAEBEugNhqromv01VD6tq/mM1rwC9/IjRVLJhnhb88sImTFu2lW3ffe92OMaYCuRPolgNtBeR1iJS\nA+9BPzF/oqpmqmq0qsapahywEhikqklQcMYxDO+9jXP95L6FiPj+as4gYHMptsdUEhHhqSFdiaoZ\nxm8XJJOdk+d2SMaYClJiolDVHGACsAzvQXuRqm4UkSdEZJAf6xgApKtqWiHThvPTM42JIrJRRNbh\nfYpqrB/rMC6IrlOTqTd1ZdP+4zz/6Ta3wzHGVBAJhufhPR6PJiUluR1GtfXg4nX8Y206i+/pT69W\nDd0OxxjjJxFZo6qekvrZm9mmzP54fTzN6kcyadE6Tp7JcTscY0w5s0RhyiwqIpxnh3Vn95EsnvzQ\nbikZE2wsUZhy0bdNI8Zf3Ia3Vu3msy3fuR2OMaYcWaIw5WbSlR3odEEUDy1Zz5GT9nvbxgQLSxSm\n3NQMC+W54Qlknspm8jvrrXCgMUHCEoUpV/HN6jLpio58tOEA73xjhQONCQaWKEy5Gz+gDb3jGvDY\nUiscaEwwsERhyl1oiPDssATyVPndonVWONCYKs4ShakQLRvV4tHr4lmRdpg5X+10OxxjTBlYojAV\nZkTvWH55YWOe/niLFQ40pgqzRGEqjIgw9aZuRNUM4/6FVjjQmKrKEoWpUDFRNfnzTV3ZuO84L3y6\n3e1wjDHnwRKFqXBXdb6Aob1a8NLyVNbsOup2OMaYUrJEYSrFY9fH07ReJA8sSiYr2woHGlOVWKIw\nlSIqIpxnh3dn15EsnvzACgcaU5VYojCVpl+bRtz589bMX7Wbz7cedDscY4yfLFGYSvXAlR3p2CSK\nh5akWOFAY6oISxSmUkWEhzJ9RALHsrL5w7tWONCYqsCvRCEiA0Vkq4ikisjDxfQbKiIqIh5nfLSI\nJPt88kQkQUSizmk/JCJ/deapKSILnXWtEpG48thQEzjim9Xl/is68OH6A7ybbIUDjQl0JSYKEQkF\nZgBXA/HAKBGJL6RfFDARWJXfpqrzVTVBVROAMcBOVU1W1e/z251pu4B/OrPdARxV1XbAdODpsm2i\nCUR3D2iLp1UD/rh0I/uscKAxAc2fM4o+QKqqpqlqNrAAGFxIvynANOB0EcsZBbx9bqOItAcaA184\nTYOBec7wEuByERE/4jRVSGiI8Ozw7uTmKb9bbIUDjQlk/iSK5sAen/F0p62AiPQAYlX1/WKWM4JC\nEgXeBLJQf7hYXbA+Vc0BMoFGfsRpqphWjWrz6HXxfLXjMHOtcKAxAcufRFHYt/mCr38iEoL3EtED\nRS5ApC+QpaobCpk8kh8nkGLX57PM8SKSJCJJGRkZRa3aBLiRvWO5vJO3cOB2KxxoTEDyJ1GkA7E+\n4y2AfT7jUUAXYLmI7AT6AYn5N7Qd5yYDAESkOxCmqmsKW5+IhAH1gCPnzquqs1TVo6qemJgYPzbD\nBCIRYeqQrtSuGcb9i6xwoDGByJ9EsRpoLyKtRaQG3oN+Yv5EVc1U1WhVjVPVOGAlMEhVk6DgjGMY\n3nsb5yrsvkUicJszPBT4TO0ZyqDWOCqCP9/YlQ17j/PiZ1Y40JhAU2KicO4TTACWAZuBRaq6UUSe\nEJFBfqxjAJCuqmmFTBvOTxPFa0AjEUkFJgFFPo5rgsfALhcwpGcLZnyeytrdVjjQmEAiwfBl3ePx\naFJSktthmDI6fvosV//1C8JDhQ9/czG1aoS5HZIxAS0nN4+w0PN/b1pE1qiqp6R+9ma2CRh1I8L5\nyzBv4cA/f2iFA40pzt5jp7j2hf/x+ZaKr5tmicIElP5tG3HHz1rz5srdLLfCgcYUKi3jBMP+/hX7\nMk8RFVHxZ96WKEzA+d1VHenQpA4PLUnhqBUONOZHNu7LZPjMFZzJyWPB+H544hpW+DotUZiAExEe\nynPDEzialc0f3t1ghQONcazZdYSRs1YSHhrConv607lZvUpZryUKE5C6NK/Hb3/ZgQ/W72dp8r6S\nZzAmyH2xPYNbXv2a6Do1WXxPf9rG1Km0dVuiMAHrnkva0qtVAx5dusEKB5pq7eMN+7ljbhKtGtVi\n0d39adGgVqWu3xKFCVihIcJzTuHAB5dY4UBTPS1O2sN989fSpXldFo7vT0xUzUqPwRKFCWitGtXm\nD9fG82XqYeat2Ol2OMZUqtn/+5YHl6RwUdto3rijL/VqhbsShyUKE/BG9YnlF50a89RHW0g9aIUD\nTfBTVZ7/ZDtPvL+Jqzo34bWxHmrXdO8FVEsUJuCJCE8N6UqtGqHcv3AdZ3OtcKAJXqrKkx9sZvon\n27xlbW7uSc2wUFdjskRhqoTGURFMvakr6/dm8uKnVjjQBKfcPOXhf6zn1f99y9iL4nhmaLcylego\nL+5HYIyfBnZpyk09mzNj+Q6+scKBJshk5+Qx8e1vWJi0h4mXt+ex6+MJCQmMH/e0RGGqlMcHdeaC\nuhFMWrSOrOwct8Mxplycys7lrteT+GD9fv5w7YVMuqIDgfQL0JYoTJVSNyKcZ4Z149tDJ5n64Ra3\nwzGmzI6fPsuts1fx3+0ZPD2kK3de3MbtkH7CEoWpci5qG80dP2/NGyt38Z9t9jO4puo6dOIMo2at\nJHnPMV4c1YMRvVu6HVKhLFGYKunBqzrSvnEdHly8zgoHmipp37FTDJ+5gh0ZJ5h1q4frujVzO6Qi\nWaIwVVJEeCjTRyRw5GQ2f1hqhQNN1fLtoZMMe3kFGcfP8Pq4vlzWsbHbIRXLEoWpsro0r8f9V3Tg\ng5T9JK6zwoGmati8/zjDXl7BqbO5vD2+H31aV3yZ8LKyRGGqtLsHtKFny/o8+u4G9mda4UAT2Nbs\nOsqImSsICxEW3d2fLs0rp0x4WfmVKERkoIhsFZFUEXm4mH5DRURFxOOMjxaRZJ9PnogkONNqiMgs\nEdkmIltEZIjTPlZEMnzmubM8NtQEp7DQEJ4bnkBOnvLg4hQrHGgC1v+2H2LMa6toWLsGi+/pT7vG\nlVcmvKxKTBQiEgrMAK4G4oFRIhJfSL8oYCKwKr9NVeeraoKqJgBjgJ2qmuxMngwcVNUOznL/47O4\nhfnzqeqr57ltppqIi67N5Gsv5H+ph3h9xU63wzHmJ5ZtPMC4uatp2bAWi+7pT2zDyi0TXlb+nFH0\nAVJVNU1Vs4EFwOBC+k0BpgGni1jOKOBtn/FxwFQAVc1T1UN+R23MOW7u05LLOsYw9aMtpB484XY4\nxhT459p07pu/ls7N67JgfD8aR0W4HVKp+ZMomgN7fMbTnbYCItIDiFXV94tZzgicRCEi9Z22KSKy\nVkQWi0gTn75DRCRFRJaISKwfMZpqTkR4ekg3atUIZdKiZCscaALCvK92MmnROvq1acibd/Slfq0a\nbod0XvxJFIW9R15wIVhEQoDpwANFLkCkL5ClqhucpjCgBfClqvYEVgB/caa9B8SpajfgE2BeEcsc\nLyJJIpKUkWEvXRloXDeCJ2/sSkp6Ji9+lup2OKYaU1X+9tl2HkvcyBXxTXjttt6ulgkvK38SRTrg\n+62+BeD7LGIU0AVYLiI7gX5AYv4NbcdIfnzZ6TCQBbzjjC8GegKo6mFVPeO0vwL0KiwoVZ2lqh5V\n9cTExPixGaY6uKZrU27q0ZwZn6eSvOeY2+GYakhVmfrRFv7yr23c2KM5L43uSUS4u2XCy8qfRLEa\naC8irUWkBt6DfmL+RFXNVNVoVY1T1ThgJTBIVZOg4IxjGN57G/nzKN4zh0udpsuBTU7/pj7rHgRs\nPr9NM9XV44M70ySqJpMWJnMqO9ftcEw1kpunPPLP9cz6bxq39m/Fs8O6Ex4AZcLLqsQtUNUcYAKw\nDO9Be5GqbhSRJ0RkkB/rGACkq2raOe2/Bx4XkRS8T0TlX7qaKCIbRWQd3qeoxvq3KcZ41Y0I5y/D\nupN26CRTP7LvGaZyZOfkMXHBNyxYvYcJl7XjT4M6B0yZ8LKSYCh94PF4NCkpye0wTIB54r1NzP7y\nW14f14cBHezypKk4p7JzuXf+GpZvzeCRqztx9yVt3Q7JLyKyRlU9JfWr+udExhThoYEdade4Dg8u\nWcexLCscaCrG8dNnuW321/xnWwZTb+paZZJEaViiMEErIjyUv45I4PCJbB5dutHtcEwQOnIym5tf\nWcna3Ud5YWQPRvUJzDLhZWWJwgS1Ls3r8ZvL2/Peun0sTd7rdjgmiBzIPM3wmSvY/t0JXrnVw/Xd\nA7dMeFlZojBB795L29LDCgeacrTz0EmGvvwVBzJP8/q4PlzWKbDLhJeVJQoT9MJCQ5g+PIGzucpD\nS6xwoCmbLQeOM2zmCk6eyeHtu/rRt00jt0OqcJYoTLWQXzjwi+2HeGPlLrfDMVXUN7uPMmLmSkIE\nFt3dn64tqkaZ8LKyRGGqjdF9W3JpxximfrSZHRlWONCUzlephxj96irqRYaz5J6LaN8kyu2QKo0l\nClNtiAjThnQjIjyUSQutcKDx3782HmDs3NW0aBDJkipYJrysLFGYaqVx3QievKEr69IzmfG5FQ40\nJXvnm3Tunb+WC5vWZeH4/jSuW/XKhJeVJQpT7VzbrSk3JDTjxc+scKAp3hsrdnL/wnX0iWvI/Dv7\n0qB21SwTXlaWKEy19KfBXWhshQNNEVSVGZ+n8ujSjfzywsbMub03dapwmfCyskRhqqV6kT8UDnzK\nCgcaH6rKUx9v4ZllW7khoRl/v6VXlS8TXlaWKEy19bN20dz+szjmrdjFF9vtx6+Mt0z45Hc3MPM/\nadzSryXPDU8IijLhZWV7wFRrvx/YyVs4cHEKmVln3Q7HuOhsbh73L0zmrVW7ue/StkwZ3CVoyoSX\nlSUKU61FhIcyfXgCh06c4dGlG0qewQSl02dzufuNNSSu28fvB3bioYGdELEkkc8Shan2uraox8TL\n25O4bh+J6/aVPIMJKt87ZcI/33qQ/3dDF+69NPjKhJeVJQpjgPsubUtCbH3+8M56DmSedjscU0mO\nnMxm9KurSNp1lL+OSOCWfq0E5OYJAAANy0lEQVTcDikgWaIwBm/hwOeGd+dsrvLgknUEwy8/muId\nyDzNiJkr2HLge2be0ovBCc3dDilgWaIwxtEmpg7/Z4UDq4Xdh7MYNvMr9h07xbzb+/DL+CZuhxTQ\n/EoUIjJQRLaKSKqIPFxMv6EioiLiccZHi0iyzydPRBKcaTVEZJaIbBORLSIyxGmvKSILnXWtEpG4\nsm+mMf65pW9LLukQw58/tMKBwWrrge8Z+vJXfH86h7fu6kf/tsFfJrysSkwUIhIKzACuBuKBUSIS\nX0i/KGAisCq/TVXnq2qCqiYAY4CdqprsTJ4MHFTVDs5y/+O03wEcVdV2wHTg6fPdOGNKS0SYNtQp\nHLhoHTlWODCoJO85xohZKwBvmfDusfVdjqhq8OeMog+QqqppqpoNLAAGF9JvCjANKOpO4CjgbZ/x\nccBUAFXNU9VDTvtgYJ4zvAS4XOw5NVOJmtSN4P/d0IV1e44x4/MdbodjyslXOw4x+pWVREWEseSe\ni+hQjcqEl5U/iaI5sMdnPN1pKyAiPYBYVX2/mOWMwEkUIpKfxqeIyFoRWSwi+RcJC9anqjlAJmDn\nhqZSXdetGYMTmvHCZ9tZZ4UDq7xPNn3H2DmraVY/kiX3XETLRtWrTHhZ+ZMoCvs2X/BIiIiE4L1E\n9ECRCxDpC2Spav4bTWFAC+BLVe0JrAD+4s/6fJY5XkSSRCQpI8PKL5jy98SgLsTUqcn9i6xwYFW2\nNHkv97y5hk4XRLHo7v40qYZlwsvKn0SRDsT6jLcAfN9KigK6AMtFZCfQD0jMv6HtGMmPLzsdBrKA\nd5zxxUDPc9cnImFAPeDIuUGp6ixV9aiqJyYmxo/NMKZ06tVyCgdmnOTpj7e4HY45D2+u3MVvFybT\nq1WDal0mvKz8SRSrgfYi0lpEauA96CfmT1TVTFWNVtU4VY0DVgKDVDUJCs44huG9t5E/jwLvAZc6\nTZcDm5zhROA2Z3go8JnaQ+3GJT9vH83Yi+KY+9VOKxxYxfx9+Q7+8O4GLuvYmHnj+hAVEe52SFVW\niYnCuU8wAVgGbAYWqepGEXlCRAb5sY4BQLqqpp3T/nvgcRFJwftEVP6lq9eARiKSCkwCinwc15jK\n8PDVnWgbU9sKB1YRqsrTH2/h6Y+3cH33ZswcY2XCy0qC4cu6x+PRpKQkt8MwQSwl/Rg3vfQV13Zr\nyvMje7gdjilCXp7y6NINzF+1m5v7tmTK4C6EWgXYIonIGlX1lNTP3sw2xg/dWtTn179oz9Lkfbyf\nYoUDA9HZ3DzuX5TM/FW7ufuSNjx5gyWJ8mKJwhg//eqytnSPrc/kdzZY4cAAc/psLve+uYalyft4\naGBHHrn6QisTXo4sURjjp7DQEKYP786ZnFwe+keKFQ4MECfO5HD7nNV8svkgUwZ35r5L27kdUtCx\nRGFMKbSJqcPkay7kv9syeNMKB7ru6MlsRr+ykq93HmH6iO6M6R/ndkhByRKFMaV0S79WDOgQw5Mf\nbibNCge65rvjpxkxawWbD3zPy7f04sYeLdwOKWhZojCmlESEZ4Z2o2ZYKPdb4UBX7DmSxbCXV5B+\n9BRzx/bmCisTXqEsURhzHnwLB7603AoHVqbt33nLhGeeOsv8O/tyUbtot0MKepYojDlP13dvxqDu\nzXjh0+2kpFvhwMqQkn6M4TNXkKfeMuE9WjZwO6RqwRKFMWUwZXAXouvU5P6FyZw+a4UDK9LKtMPc\n/MoqatUIY/Hd/el4gZUJryyWKIwpg3q1wnlmWDd2ZJzkqY+scGBF+WzLd9w2+2ua1K3Jknv7Exdd\n2+2QqhVLFMaU0cXtYwoKB/5v+6GSZzClkrhuH+NfX0P7JnVYdHd/mtaLdDukascShTHl4PcDO9Em\npjYPLllH5ikrHFhe3lq1m98s+IaeLRvw1l39aFSnptshVUuWKIwpB5E1Qpk+PIGD35/hsaUbSp7B\nlGjmf3bwf++s55IOMcwb14e6VibcNZYojCkn3WPr8+tftOPd5H18kLLf7XCqLFXlmWVbmPrRFq7r\n1pRZYzxE1rAy4W6yRGFMOfrVZe3o3qIek99dz3fHrXBgaeXlKX9cupEZn+9gVJ9Ynh/Zgxphdphy\nW5jbARgTTMJDQ3huRALXvvAFDy5JYd7tvYO6imlennI2L4+cXCUnV8nOzSPHGc/O9f49m5vH2dw8\ncvLyh5Uc5+9Zp3/+8Jeph/hw/QHGD2jDI1d3Cup9V5VYojCmnLWNqcMjV1/IY4kbeXPVbsb0a1Vk\nX98DbcFBNC+PsznntjsH2pw8zublH2iL6+/TJ++cA3N+H7+W413vDwd+nySQp+TmlW8FXRH43ZUd\n+NVl7SxJBBBLFMZUgDH9WvHJ5u+Y8t4mXv0i7ZwDvBYcgMv7QFuYsBAhPDSEsFDv3/BQISzE+9fb\n7jMcItSuEebt86N23/7+LcfbJ79/ccv5oX9kjVDqRdpN60BjicKYChASIjw7rDvP/Xsbp87mVtqB\n9qcHeLFv5qbM/EoUIjIQeB4IBV5V1aeK6DcUWAz0VtUkERkNPOjTpRvQU1WTRWQ50BQ45Uy7UlUP\nishY4Blgr9P+N1V9tXSbZYz7GteN4Kkh3dwOw5gyKzFRiEgoMAO4AkgHVotIoqpuOqdfFDARWJXf\npqrzgfnO9K7AUlVN9plttKomFbLahao6obQbY4wxpvz589xZHyBVVdNUNRtYAAwupN8UYBpQ1DOB\no4C3zytKY4wxrvEnUTQH9viMpzttBUSkBxCrqu8Xs5wR/DRRzBGRZBF5VH58IXWIiKSIyBIRifUj\nRmOMMRXEn0RR2J2wgkc1RCQEmA48UOQCRPoCWarqW9tgtKp2BS52PmOc9veAOFXtBnwCzCtimeNF\nJElEkjIyMvzYDGOMMefDn0SRDvh+q28B7PMZjwK6AMtFZCfQD0gUEY9Pn5Gcczahqnudv98Db+G9\nxIWqHlbVM063V4BehQWlqrNU1aOqnpiYGD82wxhjzPnwJ1GsBtqLSGsRqYH3oJ+YP1FVM1U1WlXj\nVDUOWAkMyr9J7ZxxDMN7bwOnLUxEop3hcOA6YIMz3tRn3YOAzWXYPmOMMWVU4lNPqpojIhOAZXgf\nj52tqhtF5AkgSVUTi18CA4B0VU3zaasJLHOSRCjeS0yvONMmisggIAc4AowtzQYZY4wpX6Ja8W+G\nVjSPx6NJSYU9ZWuMMaYoIrJGVT0l9guGRCEiGcCu85w9GgjEnyWzuErH4iq9QI3N4iqdssTVSlVL\nvMkbFImiLEQkyZ+MWtksrtKxuEovUGOzuEqnMuKyQu/GGGOKZYnCGGNMsSxRwCy3AyiCxVU6Flfp\nBWpsFlfpVHhc1f4ehTHGmOLZGYUxxphiBXWiEJGBIrJVRFJF5OFCpo8VkQynMGGyiNzpM+02Ednu\nfG4LoLhyfdpLetmxXONy+gwXkU0islFE3vJpd21/lRCXa/tLRKb7rHubiBzzmebmv6/i4nJzf7UU\nkc9F5BunKOg1PtMecebbKiJXBUJcIhInIqd89tfLlRxXKxH51IlpuYi08JlWvv++VDUoP3jf+N4B\ntAFqAOuA+HP6jMX7w0jnztsQSHP+NnCGG7gdlzPthIv7qz3wTf6+ABoHyP4qNC6399c5/X+Nt6qB\n6/urqLjc3l94r7Xf6wzHAzt9htfhrejQ2llOaADEFQdscHF/LQZuc4Z/AbxRUf++gvmMwt/f0SjM\nVcC/VfWIqh4F/g0MDIC4KpI/cd0FzHD2Cap60Gl3e38VFVdFKu1/R9/fY3F7fxUVV0XyJy4F6jrD\n9fih+OhgYIGqnlHVb4FUZ3lux1WR/IkrHvjUGf7cZ3q5//sK5kRR4u9oOAr77Qt/563suAAixFte\nfaWI3FBOMfkbVwegg4h86ax/YCnmdSMucHd/Ad5LBHi/CX9W2nkrOS5wd389DtwiIunAh3jPdvyd\n1424AFo7l6T+IyIXl1NM/sa1DhjiDN8IRIlIIz/nLZVgThTF/o6Go6jfvvBnXjfiAmip3rcwbwb+\nKiJtKzGuMLyXeS7F+030VRGp7+e8bsQF7u6vfCOBJaqaex7zllZZ4gJ399coYK6qtgCuAd4Qb/Vp\nt/dXUXHtx7u/egCTgLdEpC7lw5+4fgdcIiLfAJcAe/EWUy33/RXMiaKk39FAi/7tixLndSkuVHWf\n8zcNWA70qKy4nD5LVfWscwlgK94DtKv7q5i43N5f+c79PRa391dRcbm9v+4AFjnrXwFE4K1j5Pb+\nKjQu51LYYad9Dd57Ch0qKy5V3aeqNzmJarLTlunnNpVORdyICYQP3m+ZaXhPrfNvBnU+p09Tn+Eb\ngZX6w82gb/HeCGrgDDcMgLgaADWd4WhgO8XcqKyAuAYC83zWvwdoFAD7q6i4XN1fTr+OwE6cd5YC\n4d9XMXG5/e/rI2CsM3wh3oObAJ358c3sNMrvZnZZ4orJjwPvTee9lfzvPhoIcYafBJ6oqH9fZd6g\nQP7gPU3chjfTT3bansD7w0oAU4GNzn+Ez4FOPvOOw3vTLBW4PRDiAi4C1jvt64E7KjkuAZ4DNjnr\nHxkg+6vQuNzeX87448BThczr2v4qKi639xfem7NfOutPBq70mXeyM99W4OpAiAvv/YH8/0/XAtdX\nclxD8SbzbcCrOEm+Iv592ZvZxhhjihXM9yiMMcaUA0sUxhhjimWJwhhjTLEsURhjjCmWJQpjjDHF\nskRhjDGmWJYojDHGFMsShTHGmGL9f34ebtSeialDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x28296f3b7b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "x_axis = subsample_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  12 out of  15 | elapsed: 44.2min remaining: 11.0min\n",
      "[Parallel(n_jobs=4)]: Done  15 out of  15 | elapsed: 53.5min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "       error_score='raise',\n",
       "       estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt', class_weight=None,\n",
       "        colsample_bytree=1.0, importance_type='split', learning_rate=0.1,\n",
       "        max_bin=127, max_depth=7, min_child_samples=30,\n",
       "        min_child_weight=0.001, min_split_gain=0.0, n_estimators=428,\n",
       "        n_jobs=4, num_class=9, num_leaves=70, objective='multiclass',\n",
       "        random_state=None, reg_alpha=0.0, reg_lambda=0.0, silent=False,\n",
       "        subsample=0.8, subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid=True, n_jobs=4,\n",
       "       param_grid={'colsample_bytree': [0.5, 0.6, 0.7, 0.8, 0.9]},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#调整sub_feature\n",
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':70,\n",
    "          'min_child_samples':30,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'subsample': 0.8,\n",
    "          'bagging_freq': 1,\n",
    "          #'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "colsample_bytree_s = [i/10.0 for i in range(5,10)]\n",
    "tuned_parameters = dict( colsample_bytree = colsample_bytree_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.47398601959122055\n",
      "{'colsample_bytree': 0.6}\n"
     ]
    }
   ],
   "source": [
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAD8CAYAAABpcuN4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xt8VfWZ7/HPk4RwDbdcACGQAAFF\nQZAIohVRtFWr0FZUqMdKb3bmDGJn6rzGns5MrXamjtOO1erYUWtrrXfP6TFQ59iKBq1VJFhUQElC\nCBJACIR7CLk954+9EjdpLjuQZGfv/X2/XvvFXr/1W2s9a5Hk2c/6rbW2uTsiIiJtSYp2ACIi0rsp\nUYiISLuUKEREpF1KFCIi0i4lChERaZcShYiItEuJQkRE2qVEISIi7VKiEBGRdqVEO4CukJGR4Tk5\nOdEOQ0Qkpqxbt26vu2d21C8uEkVOTg5FRUXRDkNEJKaY2bZI+unUk4iItEuJQkRE2qVEISIi7VKi\nEBGRdilRiIhIu5QoRESkXUoUIiLSLiUKEekVqo7W8qs3t/J6cSU1dQ3RDkfCxMUNdyISu+oaGvnN\n29u49w/FHKqpB6B/n2TmTEhn3uRM5k3KYmz6gChHmdiUKEQkal4vruTOlZso3XOEC/My+IfLT6fy\n8HEKN++hsLiSVz/aA2xkfMZALpqcybzJWczOHU6/PsnRDj2hRJQozOxy4D4gGXjU3e9uo98i4Hng\nXHcvMrMbgL8P6zINOAfYArwR1j4G+I27f9vMlgL/DuwI5j3g7o9Gvksi0ttt3XuUf/ndJl75cA/j\n0gfwyFfyufSMLMwMgItPz2ruV7h5D4WbK3lqzcf88s1y+vVJ4vwJGao2epC5e/sdzJKBYuAyoAJY\nCyxx900t+qUBvwNSgWXuXtRi/lTgRXcf38o21gF/6+6vB4ki392XRboT+fn5rmc9ifR+h2vqeOC1\nUh7741ZSk5O4ZX4eX70gh74pHVcINXUNvFW2j9WbKyncvIfyfdUAqjZOgZmtc/f8jvpFUlHMAkrd\nvSxY8TPAQmBTi353AfcAt7WxniXA060EmgdkcWKFISJxpLHReeHdCu75f5vZe+Q4184cw99fPpms\ntH4Rr6Nfn2QunpzFxZOzgDMpb6o2ik+sNuaMT2fe5CzmTc5kXPrA7tupBBJJohgNbA+brgBmh3cw\nsxlAtruvNLO2EsX1hBJMS0uAZ/3E0uYaM5tLqJL5W3ff3spyIhID1m2r4o6CTXyw4yDnjB3KL27K\n5+zsoae83pyMgSzNyGXpBbnU1DXwdtk+CjdXsrq4ku8XbAQgN2MgF03KZN7kTM4bn65q4yRFkiis\nlbbmP+pmlgTcCyxtcwVms4Fqd9/QyuzFwI1h0yuAp939uJn9FfA4cEkr67wZuBlg7NixHe+FiPSo\nXQePcfd/f8SL63cyYnBffnr9dBZOP615HKIr9euTHFQRobGNbfuOUhiconr6nY/51Z/K6ZuSFLqS\nalLoNFVOhqqNSEUyRjEHuMPdPxdMfxfA3X8UTA8hNDh9JFhkJFAFLGgapzCze4FKd//XFus+G3je\n3Se1se1koMrdh7QXo8YoRHqPmroGHn69jIcKt9DgzrfmjuevLprAwL7RuciyZbWxde9RAHLSBzBv\nchYXTc5kToJWG105RrEWyDOzXEJXIi0Gvtw0090PAhlhGy4EbgtLEknAtcDcVtb9F+MWZjbK3XcF\nkwuADyOIUUSizN357w2f8C+/+5AdB45x5dSRfPeKM8geHt2rktqrNp5Z+2m1cd744L6NyVnkqto4\nQYeJwt3rzWwZ8DKhy2Mfc/eNZnYnUOTuBR2sYi5Q0TQY3sJ1wJUt2pab2QKgnlBlsrSjGEUkujbt\nPMQPVmxkzdYqTh+ZxtPfPI85E9KjHVarxqUP5KbzB3LT+TnU1DWwZmsVhZv3sHpzJT9YsYkfrNjE\nuPQBzaeozhufTv/UxKs2wnV46ikW6NSTSHTsO3Kcn/yhmGfe+Zgh/fvwnc9OZvG52aQkx+bTgT7e\nV01hcei+jT9t2UtNXSN9U5KYPb5pbCOT3IyB3TLOEg2RnnpSohCRTqtraOSJt7bx01eKOVrbwFfm\njOPb8ycxZECfaIfWZWrqGnhna1XoNFXxHsoqQ2MbY4cPCE5RZTJnfEZMVxtKFCLSLVYXV3Lnio1s\nqTzKhXkZ/PNVU8gbkRbtsLrd9qrq5rvE/7RlH8fqGkhNSWJ27vDm+zbGx1i1oUQhIl1q696j/HDl\nJlZ9tIec9AH801VTuOT0rJj6w9hVauoaWFte1TwoviWoNrKH92fepCwuPj02qg0lChHpEodq6njg\n1VJ++eZW+qYkc8slE1ka4WM3EsX2qmoKiytZvXkPb5bGTrWhRCEip6Sx0Xl+3Xb+/eXN7Dtay7Uz\nx3Db5zr32I1EdLy+gbVb9zc/XqR0T+gWs6ZqY97kTOZMSGdAavQf3q1EISInrai8ih+sCD12Y+a4\nYXz/6ilMG3Pqj91IRNurqlldXNl8JVV1bQOpyUnMHj88eLxIFhMyo1NtKFGISKftPBB67EbBezsZ\nObgf373ydBac3T2P3UhEbVUbY4b1b35s+vkTe67aUKIQkYjV1DXwX6vLeGh1Ke6EHrsxb0KvOD0S\nzyr2VwcD4idWG7Nyhzdfgjshc1C3JWolChHpkLvz0gef8K8vhR678fmpo7j9itOj/tiNRHS8voGi\n8v3Nl+CWBNXG6KH9mx8tcv6E9C59ZpYShYi0a+POg/xgxSbe2VrFGaMG8/2rp3De+N752I1EVLE/\nbGyjdC9Hg2rj3NxhzYPiE7NOrdpQohCRVu07cpwf/76YZ9Z+zLABqdz22clcf242yUkah+itausb\nKSqvorA4dN9G8e5Pq43vXnk6V0077aTW25VPjxWROFBb38iv3yrnvlUlHKtt4Kvn53LrpXkM6R8/\nj92IV6kpSZw/MYPzJ2bwv648gx0HjjV/Jezwgandvn0lCpEEULh5D3eu3ERZ5VHmTsrkn686g4lZ\n8f/YjXg1emh/vjx7LF+e3TNf2qZEIRLHyiqP8MPffcirH+0hN2Mgjy3N5+LJifnYDTl5ShQicehQ\nTR0/W1XCL98sp1+fZL535RncdH4OqSmx+fhviS4lCpE40tDoPF8UeuxGVXUt183M5rbPTSYzrW+0\nQ5MYpkQhEifWllfxgxUb2bDjEPnjhvGrq2cxdUy7XzcvEhElCpEYtyN47MaK93Yyakg/7l8yg6un\njdI4hHQZJQqRGHWstoH/en0LP1+9BXe4dX4e37povB67IV0uop8oM7scuA9IBh5197vb6LcIeB44\n192LzOwG4O/DukwDzgG2AG+EtY8BfuPu3zazvsCvgZnAPuB6dy/v1F6JxDF3Z+X7u/jRSx+y82AN\nn582iu9ecTpjhumxG9I9OkwUZpYMPAhcBlQAa82swN03teiXBiwH1jS1ufuTwJPB/KnAi+6+Ppg9\nPWzZdcD/CSa/Dux394lmthj4N+D6k9s9kfiyYcdB7lyxiXfKq5gyajD3Xj+d2XrshnSzSCqKWUCp\nu5cBmNkzwEJgU4t+dwH3ALe1sZ4lwNMtG80sD8ji0wpjIXBH8P4F4AEzM4+HZ42InKS9R47zk99v\n5pm12xk2IJUffWkq1+XrsRvSMyJJFKOB7WHTFcDs8A5mNgPIdveVZtZWorieUBJoaQnwbFgiaN6e\nu9eb2UEgHdjbYps3AzcDjB3bM3cnivS05sduvFLCsboGvnZBLsvn67Eb0rMiSRStfWRp/nRvZknA\nvcDSNldgNhuodvcNrcxeDNwY6faaG9wfBh6G0EMB29q2SKx67aM93LVyE2V7jzJvcib/+PkpTMwa\nFO2wJAFFkigqgOyw6THAzrDpNOAsoDC4HG8kUGBmC9y96ZGui2n9tNPZQIq7r2tlexVmlgIMAaoi\n2x2R2Lel8gg/XLmJ1zZXMj5jIL9cei4Xn54V7bAkgUWSKNYCeWaWC+wg9Ef/y00z3f0gkNE0bWaF\nwG1NSSKoOK4F5ray7tbGLQqAm4C3gEXAqxqfkERw8FjosRu/+lM5/fsk84+fP4OvzNFjNyT6OkwU\nwTjBMuBlQpfHPubuG83sTqDI3Qs6WMVcoKJpMLyF64ArW7T9AnjCzEoJVRKLO4pRJJY1NDrPFW3n\nx8FjNxafm813PjuZjEF67Ib0DvriIpEoWlO2jx+s2MSmXYc4N2cY37/6TM4arcduSM/QFxeJ9GI7\nDhzjX1/6kN+9v4vThvTjZ0tmcJUeuyG9lBKFSA86VtvAz1eHHrthBt++NI9vzZ1A/9TkaIcm0iYl\nCpEe4O6sCB67setgDVdNG8V3rzyD0UP7Rzs0kQ4pUYh0sw07DnJHwUaKtu3nzNMGc9/iGczKHR7t\nsEQipkQh0k1q6hq4o2AjzxZtZ/iAVO7+0lSu1WM3JAYpUYh0k1/8cSvPrN3ONz6Ty/JL8xjcT4/d\nkNikRCHSDY4er+fRN8q4eHIm/3jVlGiHI3JKdMunSDf49Vvb2F9dx62XTop2KCKnTIlCpIsdPV7P\nI2+UcdGkTKZnD412OCKnTIlCpIs9uWYbVUdrWT4/L9qhiHQJJQqRLnSstoGHXy/jwrwMZo4bFu1w\nRLqEEoVIF3pyzTb2HqnlVlUTEkeUKES6SOjxHGVcMDGd/BzdUCfxQ4lCpIs89c7H7D1ynFvn60on\niS9KFCJdoKYu9LC/OePT9XgOiTtKFCJd4Jl3Pqby8HFd6SRxSYlC5BTV1DXw0OotzModzpwJ6dEO\nR6TLKVGInKLniraz+9Bxvq1qQuJURInCzC43s81mVmpmt7fTb5GZuZnlB9M3mNn6sFejmU0P5qWa\n2cNmVmxmH5nZNUH7UjOrDFvmG12xoyLd4Xh9Aw8VbuHcnGGqJiRudfhQQDNLBh4ELgMqgLVmVuDu\nm1r0SwOWA2ua2tz9SeDJYP5U4EV3Xx/M/h6wx90nmVkSED4C+Ky7Lzv53RLpGc8XVbDrYA3/vuhs\nfY2pxK1IKopZQKm7l7l7LfAMsLCVfncB9wA1baxnCfB02PTXgB8BuHuju++NOGqRXqC2vpGHCrdw\nztihXDBR1YTEr0gSxWhge9h0RdDWzMxmANnuvrKd9VxPkCjMrOlJaXeZ2btm9ryZjQjre42ZvW9m\nL5hZdgQxivS4F9ZVsOPAMW69dJKqCYlrkSSK1n4DvHlm6LTRvcB32lyB2Wyg2t03BE0pwBjgTXc/\nB3gL+HEwbwWQ4+7TgFeAx9tY581mVmRmRZWVlRHshkjXqa1v5MHXSpmePZS5eRnRDkekW0WSKCqA\n8E/1Y4CdYdNpwFlAoZmVA+cBBU0D2oHFnHjaaR9QDfw2mH4eOAfA3fe5+/Gg/RFgZmtBufvD7p7v\n7vmZmZkR7IZI1/ntn5uqiTxVExL3IkkUa4E8M8s1s1RCf/QLmma6+0F3z3D3HHfPAd4GFrh7ETRX\nHNcSGttoWsYJVQ7zgqb5wKag/6iwbS8APjy5XRPpHnUNjTzwWinTxgxh3iR9SJH41+FVT+5eb2bL\ngJeBZOAxd99oZncCRe5e0P4amAtUuHtZi/Z/AJ4ws58ClcBXg/blZrYAqAeqgKUR741ID/jtn3ew\nveoYd1x9pqoJSQgW+nAf2/Lz872oqCjaYUgCqG9oZP5/rGZwvz4ULLtAiUJimpmtc/f8jvrpzmyR\nTvi/63eybV81y+drbEIShxKFSITqG0JXOk0ZNZhLz8iKdjgiPUaJQiRCK97fyda9R1VNSMJRohCJ\nQEOj87NXSzl9ZBqfnTKi4wVE4ogShUgEVr6/k7LKo9w6P4+kJFUTkliUKEQ60NDo3L+qhMkj0vjc\nmSOjHY5Ij1OiEOnASx/sYktlaGxC1YQkIiUKkXY0Njo/e7WEvKxBXHGWqglJTEoUIu347w2fULz7\nCLeompAEpkQh0obGYGxiQuZAPj91VMcLiMQpJQqRNvx+0yds3n2Y5fPzSFY1IQlMiUKkFY2Nzn2r\nShmfMZCrpp0W7XBEokqJQqQVf/hwNx/uOsSySyaqmpCEp0Qh0oJ7aGwiJ30AC85WNSGiRCHSwisf\n7mHjzkMsuySPlGT9iojot0AkTFM1MXb4AL4wXdWECChRiJzgtc17+GDHQZZdPFHVhEhAvwkiAXfn\nvldKyB7eny+eMzra4Yj0GhElCjO73Mw2m1mpmd3eTr9FZuZmlh9M32Bm68NejWY2PZiXamYPm1mx\nmX1kZtcE7X3N7NlgW2vMLOfUd1OkY4XFlbxXcZC/mTeRPqomRJp1+NtgZsnAg8AVwBRgiZlNaaVf\nGrAcWNPU5u5Puvt0d58O3AiUu/v6YPb3gD3uPilY7+qg/evAfnefCNwL/NvJ7pxIpJqqidFD+/Ol\nc8ZEOxyRXiWSj02zgFJ3L3P3WuAZYGEr/e4C7gFq2ljPEuDpsOmvAT8CcPdGd98btC8EHg/evwDM\nN32dmHSzN0r2sn77Af7m4omkpqiaEAkXyW/EaGB72HRF0NbMzGYA2e6+sp31XE+QKMxsaNB2l5m9\na2bPm1nT14Y1b8/d64GDQHoEcYqcFHfnvlUlnDakH4tmqpoQaSmSRNHap3lvnmmWROgU0XfaXIHZ\nbKDa3TcETSnAGOBNdz8HeAv4cSTbC1vnzWZWZGZFlZWVEeyGSOveLN3Hum37+WtVEyKtiuS3ogLI\nDpseA+wMm04DzgIKzawcOA8oaBrQDizmxNNO+4Bq4LfB9PPAOS23Z2YpwBCgqmVQ7v6wu+e7e35m\nZmYEuyHyl0LVRDGjhvTjunxVEyKtiSRRrAXyzCzXzFIJ/dEvaJrp7gfdPcPdc9w9B3gbWODuRdBc\ncVxLaGyjaRkHVgDzgqb5wKbgfQFwU/B+EfBq0F+ky721ZR9ry/fz1/Mm0DclOdrhiPRKKR11cPd6\nM1sGvAwkA4+5+0YzuxMocveC9tfAXKDC3ctatP8D8ISZ/RSoBL4atP8iaC8lVEksjnx3RDrnvlUl\njBjcl+vyszvuLJKgOkwUAO7+EvBSi7Z/bqPvvBbThYROR7Xst41QEmnZXkOoAhHpVm+X7WPN1iq+\nf/UU+vVRNSHSFo3cScK675USMtP6smTW2GiHItKrKVFIQnpnaxVvle3jry6aoGpCpANKFJKQ7l9V\nQsagvnxZ1YRIh5QoJOGs21bFH0v38q254+mfqmpCpCNKFJJwfvpKCekDU7nhPFUTIpFQopCE8u7H\n+3mjZC83zx3PgNSILvoTSXhKFJJQ7nulhOEDU7lxzrhohyISM5QoJGGs336A1cWVfPNCVRMinaFE\nIQnj/lUlDB3QR9WESCcpUUhCeL/iAK9+tIdvXjieQX1VTYh0hhKFJIT7V5UwpH8fvqJqQqTTlCgk\n7m3YcZBXPtzDNz6TS1q/PtEORyTmKFFI3Lt/VQmD+6Vw0wU50Q5FJCYpUUhc27TzEL/ftJuvfSaX\nwaomRE6KEoXEtftXlZDWL4WvXpAb7VBEYpYShcStD3cd4v9t/ISvXpDLkP6qJkROlhKFxK0HXi1l\nUN8Uvq5qQuSUKFFIXCrefZiXNuxi6fk5DBmgakLkVChRSFy6f1UJA/ok8/XPqJoQOVURJQozu9zM\nNptZqZnd3k6/RWbmZpYfTN9gZuvDXo1mNj2YVxiss2leVtC+1Mwqw9q/0RU7KomjZPdhfvfBLm46\nP4dhA1OjHY5IzOvwWQZmlgw8CFwGVABrzazA3Te16JcGLAfWNLW5+5PAk8H8qcCL7r4+bLEb3L2o\nlc0+6+7LOrszIgA/e7WU/n2S+caF46MdikhciKSimAWUunuZu9cCzwALW+l3F3APUNPGepYAT59U\nlCIRKt1zhBXv7+Qrc3IYrmpCpEtEkihGA9vDpiuCtmZmNgPIdveV7aznev4yUfwyOL30T2ZmYe3X\nmNn7ZvaCmWW3tjIzu9nMisysqLKyMoLdkETw4Gul9EtJ5psXamxCpKtEkiislTZvnmmWBNwLfKfN\nFZjNBqrdfUNY8w3uPhW4MHjdGLSvAHLcfRrwCvB4a+t094fdPd/d8zMzMyPYDYl3ZZVHeHH9Dm6c\nM470QX2jHY5I3IgkUVQA4Z/qxwA7w6bTgLOAQjMrB84DCpoGtAOLaVFNuPuO4N/DwFOETnHh7vvc\n/XjQ7RFgZqQ7I4ntgddKSU1J4psamxDpUpEkirVAnpnlmlkqoT/6BU0z3f2gu2e4e4675wBvAwua\nBqmDiuNaQmMbBG0pZpYRvO8DXAVsCKZHhW17AfDhKeyfJIjyvUd5cf1O/sfscWSmqZoQ6UodXvXk\n7vVmtgx4GUgGHnP3jWZ2J1Dk7gXtr4G5QIW7l4W19QVeDpJEMqFTTI8E85ab2QKgHqgClnZmhyQx\nPfhaKSlJxs0XqZoQ6Wrm7h336uXy8/O9qKi1q2wlEXy8r5qLf1LIV+aM4/tXnxntcERihpmtc/f8\njvrpzmyJeQ++VkpykvFXF02IdigicUmJQmLa9qpq/ve7FXx51lhGDO4X7XBE4pIShcS0/ywsJclU\nTYh0JyUKiVkV+6t5YV0Fi2dlM3KIqgmR7qJEITHrocItAKomRLqZEoXEpJ0HjvFc0Xauy8/mtKH9\nox2OSFxTopCY1FRN/M+LJ0Y5EpH4p0QhMeeTgzU8u3Y7i2ZmM1rVhEi3U6KQmPPz1VtodOd/ztPY\nhEhPUKKQmLL7UA1PvfMx15wzhuzhA6IdjkhCUKKQmPLz1VtoaHT+RmMTIj1GiUJixp5DNTy15mO+\nNGM0Y9NVTYj0FCUKiRkPv15GfaOz7BJVEyI9SYlCYkLl4eP8Zs02Fk4/jXHpA6MdjkhCUaKQmPDI\nG2XU1jdyyyV50Q5FJOEoUUivt/fIcZ54axsLp48mN0PVhEhPU6KQXu+RN8o4Xt+gsQmRKFGikF6t\n6mgtT7y1javPPo0JmYOiHY5IQoooUZjZ5Wa22cxKzez2dvotMjM3s/xg+gYzWx/2ajSz6cG8wmCd\nTfOygva+ZvZssK01ZpZz6rspserRN8o4VtfALaomRKKmw0RhZsnAg8AVwBRgiZlNaaVfGrAcWNPU\n5u5Puvt0d58O3AiUu/v6sMVuaJrv7nuCtq8D+919InAv8G8nuW8S4/YfreXxP5Xz+amjmJiVFu1w\nRBJWJBXFLKDU3cvcvRZ4BljYSr+7gHuAmjbWswR4OoLtLQQeD96/AMw3M4tgOYkzv/jjVqrrGlg+\nX1c6iURTJIliNLA9bLoiaGtmZjOAbHdf2c56rucvE8Uvg9NO/xSWDJq35+71wEEgveXKzOxmMysy\ns6LKysoIdkNiyYHqWn71p3KuPGsUk0aomhCJpkgSRWuf5r15plkSoVNE32lzBWazgWp33xDWfIO7\nTwUuDF43RrK95gb3h909393zMzMzO94LiSmPvVnOkeP13DJfYxMi0RZJoqgAssOmxwA7w6bTgLOA\nQjMrB84DCpoGtAOLaVFNuPuO4N/DwFOETnGdsD0zSwGGAFWR7Y7Eg4PH6vjlm1u5/MyRnD5ycLTD\nEUl4kSSKtUCemeWaWSqhP/oFTTPd/aC7Z7h7jrvnAG8DC9y9CJorjmsJjW0QtKWYWUbwvg9wFdBU\nbRQANwXvFwGvuvtfVBQSv3755lYO19RrbEKkl0jpqIO715vZMuBlIBl4zN03mtmdQJG7F7S/BuYC\nFe5eFtbWF3g5SBLJwCvAI8G8XwBPmFkpoUpicaf2SGLaoZo6HvvjVj47ZQRTTlM1IdIbdJgoANz9\nJeClFm3/3EbfeS2mCwmdjgpvOwrMbGP5GkIViCSgx98s55CqCZFeRXdmS69xuKaOR/+4lUvPyOKs\n0UOiHY6IBJQopNf49VvbOHisjlvnT4p2KCISRolCeoUjx+t55I0yLjk9i6ljVE2I9CZKFNIr/Pqt\ncg5U13GrxiZEeh0lCom6o8frefSNrcybnMnZ2UOjHY6ItKBEIVH3m7e3UXW0Vlc6ifRSShQSVdW1\n9Tz8ehkX5mVwzthh0Q5HRFqhRCFR9eTbH7PvaC3fvlTVhEhvpUQhUXOstoH/en0Ln5mYwcxxw6Md\njoi0QYlCouapdz5m75FablU1IdKrKVFIVNTUNfDz1VuYMz6dc3NUTYj0ZkoUEhVPv/MxlYePq5oQ\niQFKFNLjmqqJ2bnDOW/8X3x5oYj0MkoU0uOeK9rO7kOqJkRihRKF9Kjj9Q08VLiFc3OGMUfVhEhM\nUKKQHvVcUQW7DtZw6/xJmLX29egi0tsoUUiPOV7fwEOvlTJz3DAumKhqQiRWKFFIj3lhXQU7D9Zw\n6/w8VRMiMSSiRGFml5vZZjMrNbPb2+m3yMzczPKD6RvMbH3Yq9HMprdYpsDMNoRN32FmO8KWufJk\nd056j9r6Rv7ztS3MGDuUC/Myoh2OiHRCh9+ZbWbJwIPAZUAFsNbMCtx9U4t+acByYE1Tm7s/CTwZ\nzJ8KvOju68OW+RJwpJXN3uvuP+787khv9X/erWDHgWP88ItnqZoQiTGRVBSzgFJ3L3P3WuAZYGEr\n/e4C7gFq2ljPEuDppgkzGwT8HfDDTkUsMaeuoZEHXivl7DFDmDcpM9rhiEgnRZIoRgPbw6YrgrZm\nZjYDyHb3le2s53rCEgWhxPIToLqVvsvM7H0ze8zM9OzpGPfbd3dQsf8Yt16qsQmRWBRJomjtN9ub\nZ5olAfcC32lzBWazgWp33xBMTwcmuvtvW+n+EDABmA7sIpRMWlvnzWZWZGZFlZWVEeyGRENTNTFt\nzBAunpwV7XBE5CREkigqgOyw6THAzrDpNOAsoNDMyoHzgIKmAe3AYk6sJuYAM4P+fwQmmVkhgLvv\ndvcGd28EHiF06usvuPvD7p7v7vmZmTqd0Vu9uH4nH1dVs/wSVRMisSqSRLEWyDOzXDNLJfRHv6Bp\nprsfdPcMd89x9xzgbWCBuxdBc8VxLaGxjaZlHnL304L+nwGK3X1e0H9U2La/CGxAYlJ9QyMPvFrC\nmacNZv4ZqiZEYlWHVz25e72ZLQNeBpKBx9x9o5ndCRS5e0H7a2AuUOHuZRHGdE9wasqBcuBbES4n\nvUzBezsp31fNf904U9WESAwzd++4Vy+Xn5/vRUVF0Q5DwjQ0Opf9x2r69knmpeWfUaIQ6YXMbJ27\n53fUT3dmS7dY8d5OyvYe5dbQ5yaxAAAK/ElEQVT5E5UkRGKcEoV0uYZG52evljB5RBqfnTIy2uGI\nyClSopAu97sPdrGl8ijL5+eRlKRqQiTWKVFIl2psdH62qoRJIwZxxVmqJkTigRKFdKmXNuyiZM8R\nbrlE1YRIvFCikC4TqiZKmZg1iCunjup4ARGJCUoU0mVe3vgJm3cf5pZLJpKsakIkbihRSJdobHTu\nW1XC+MyBXDXttGiHIyJdSIlCusTvN+3mo09UTYjEow4f4RHPXvpgF88VbWfSiLTgNYiJWYMYkJrQ\nh6XT3J37V5WQmzGQq1VNiMSdhP6LWFvfyO5Dx/lT6T5qGxoBMIPsYQOYNGJQcwLJGzGICZmD6Ncn\nOcoR906vfLiHTbsO8ZNrzyYlWUWqSLxJ6ETxhRmj+cKM0dQ3NLKtqpriTw5TvPsIxXsOU/zJYQo3\nV1LfGHoWVpJBTvrA5sojb0Qak0emkZM+kNSUxP3j6O7ct6qYcekDWDhd1YRIPEroRNEkJTmJCZmh\nquGKqZ+219Y3snXvUYp3H6Zk92E27z5M8e7D/H7TJwT5g5QkY3zmQPJGpDEpK43JI0NJZNzwAQnx\n6frVj/awYcch7lk0LSH2VyQRKVG0IzUlickjQ5VDuJq6BrZUHqFk9xE2B0nkg4qD/O79XScsOyFz\n0AmnsCaNGET2sAFxcyNaqJooIXt4f744Y3THC4hITFKiOAn9+iRz5mlDOPO0ISe0V9fWU7rnCJs/\nOUzJniMU7z7M2q1VvLj+0y8E7N8nmYlZg5oTx6QRaUwamcZpQ/rF3FNWCzdX8n7FQf7tmqn0UTUh\nEreUKLrQgNQUpo0ZyrQxQ09oP1RTR8nuI5TsDsZAdh/mjZJK/ve7Fc19BvVNIW/EICZlhQbPJ48M\nVSFZaX17ZQJpqiZGD+3Pl84ZE+1wRKQbKVH0gMH9+jBz3DBmjht2QvuB6trmxNH0+sOHu3m2aHtz\nnyH9+3w6eB5cgTV5RBrpg/r29G6c4PWSvazffoB//aKqCZF4p0QRRUMHpDIrdzizcoef0L73yPFQ\n4vjkMMV7jlD8yWFWvreTp2rqm/ukD0xtThpNV2BNykpjyIA+3R63u3PfK8WMHtqfRTNVTYjEu4gS\nhZldDtxH6DuzH3X3u9votwh4HjjX3YvM7Abg78O6TAPOcff1YcsUAOPd/axgejjwLJBD6Duzr3P3\n/Z3cr5iWMagvGYP6cv6EjOY2d2fP4eNs/uRwWAVyhBfWVXC0tqG5X1ZaXyaPTCMvKxgDGZlGXtYg\n0vp1XQL5Y+le3v34AD/8wlkJfWmwSKLoMFGYWTLwIHAZUAGsNbMCd9/Uol8asBxY09Tm7k8CTwbz\npwIvtkgSXwKOtNjk7cAqd7/bzG4Ppv/hJPYtrpgZIwb3Y8TgfsydlNnc7u7sOHCMkuAUVugqrCM8\n9c42auoam/uNHto/NAZyinehh6qJEkYN6ce1+aomRBJBJH8lZgGl7l4GYGbPAAuBTS363QXcA9zW\nxnqWAE83TZjZIODvgJuB58L6LQTmBe8fBwpRomiTmTFm2ADGDBvAxadnNbc3NDoV+6tPGAPZ/Mnh\nDu9CnzQijfGZA9u8C/2tLfso2rafOxeeSd8U3akukggiSRSjge1h0xXA7PAOZjYDyHb3lWbWVqK4\nnlASaHIX8BOgukW/Ee6+C8Ddd5lZFtJpyUnGuPSBjEsfyGVTRjS3/8Vd6EESifQu9J+uKmHE4L5c\nl58drV0TkR4WSaJo7dpMb55plgTcCyxtcwVms4Fqd98QTE8HJrr735pZTifiDV/nzYSqEcaOHXsy\nq0hIkdyFHv5qeRd6faNzx9VT9NwrkQQSSaKoAMI/Po4BdoZNpwFnAYXB9f4jgQIzW+DuRUGfxYSd\ndgLmADPNrDyIIcvMCt19HrDbzEYF1cQoYE9rQbn7w8DDAPn5+d5aH4lcR3ehNw2eHz1ez+JZSswi\niSSSRLEWyDOzXGAHoT/6X26a6e4HgebLc8ysELitKUkEFce1wNywZR4CHgrm5wArgyQBUADcBNwd\n/PviyeyYdI227kIXkcTR4bWN7l4PLANeBj4EnnP3jWZ2p5ktiGAbc4GKpsHwCNwNXGZmJYSutGr1\nUlwREekZ5h77Z23y8/O9qKio444iItLMzNa5e35H/XS3lIiItEuJQkRE2qVEISIi7VKiEBGRdilR\niIhIu5QoRESkXXFxeayZVQLbTnLxDGBvF4bTVRRX5yiuzuutsSmuzjmVuMa5e2ZHneIiUZwKMyuK\n5Drinqa4OkdxdV5vjU1xdU5PxKVTTyIi0i4lChERaZcSRfAE2l5IcXWO4uq83hqb4uqcbo8r4cco\nRESkfaooRESkXXGdKMzscjPbbGalZnZ7K/OXmlmlma0PXt8Im3eTmZUEr5t6UVwNYe0FPRlX0Oc6\nM9tkZhvN7Kmw9qgdrw7iitrxMrN7w7ZdbGYHwuZF8+ervbiiebzGmtlrZvZnM3vfzK4Mm/fdYLnN\nZva53hCXmeWY2bGw4/XzHo5rnJmtCmIqNLMxYfO69ufL3ePyBSQDW4DxQCrwHjClRZ+lwAOtLDsc\nKAv+HRa8HxbtuIJ5R6J4vPKAPzcdCyCrlxyvVuOK9vFq0f8W4LHecLzaiivax4vQufa/Dt5PAcrD\n3r8H9AVyg/Uk94K4coANUTxezwM3Be8vAZ7orp+veK4oZgGl7l7m7rXAM8DCCJf9HPAHd69y9/3A\nH4DLe0Fc3SmSuL4JPBgcE9y96Wtqo3282oqrO3X2/3EJn34dcLSPV1txdadI4nJgcPB+CJ9+5fJC\n4Bl3P+7uW4HSYH3Rjqs7RRLXFGBV8P61sPld/vMVz4liNLA9bLoiaGvpmqB0e8HMmr4bPNJlezou\ngH5mVmRmb5vZF7oopkjjmgRMMrM3g+1f3olloxEXRPd4AaFTBIQ+Cb/a2WV7OC6I7vG6A/gfZlYB\nvESo2ol02WjEBZAbnJJabWYXdlFMkcb1HnBN8P6LQJqZpUe4bKfEc6KwVtpaXuK1Ashx92nAK8Dj\nnVg2GnEBjPXQXZhfBn5qZhN6MK4UQqd55hH6JPqomQ2NcNloxAXRPV5NFgMvuHvDSSzbWacSF0T3\neC0BfuXuY4ArgSfMLCnCZaMR1y5Cx2sG8HfAU2Y2mK4RSVy3AReZ2Z+Bi4AdQH2Ey3ZKPCeKCiD8\nk/gYWpSM7r7P3Y8Hk48AMyNdNkpx4e47g3/LgEJgRk/FFfR50d3rglMAmwn9gY7q8WonrmgfryaL\nOfH0TrSPV1txRft4fR14Ltj+W0A/Qs8xivbxajWu4FTYvqB9HaExhUk9FZe773T3LwWJ6ntB28EI\n96lzumMgpje8CH3KLCNUWjcNBp3Zos+osPdfBN72TweDthIaCBoWvB/eC+IaBvQN3mcAJbQzUNkN\ncV0OPB62/e1Aei84Xm3FFdXjFfSbDJQT3LPUG36+2okr2j9f/w0sDd6fQeiPmwFncuJgdhldN5h9\nKnFlNsVBaNB5Rw//3GcAScH7fwHu7K6fr1Peod78IlQmFhPK9N8L2u4EFgTvfwRsDP4TXgNOD1v2\na4QGzUqBr/aGuIDzgQ+C9g+Ar/dwXAb8B7Ap2P7iXnK8Wo0r2scrmL4DuLuVZaN2vNqKK9rHi9Dg\n7JvB9tcDnw1b9nvBcpuBK3pDXITGB5p+T98Fru7huBYRSubFwKMESb47fr50Z7aIiLQrnscoRESk\nCyhRiIhIu5QoRESkXUoUIiLSLiUKERFplxKFiIi0S4lCRETapUQhIiLt+v+T9RVWNyToMAAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x282969e9668>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "x_axis = colsample_bytree_s\n",
    "\n",
    "plt.plot(x_axis, -test_means)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 2 candidates, totalling 6 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done   3 out of   6 | elapsed:  6.4min remaining:  6.4min\n",
      "[Parallel(n_jobs=4)]: Done   6 out of   6 | elapsed: 10.0min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "       error_score='raise',\n",
       "       estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt', class_weight=None,\n",
       "        colsample_bytree=1.0, importance_type='split', learning_rate=0.1,\n",
       "        max_bin=127, max_depth=7, min_child_samples=30,\n",
       "        min_child_weight=0.001, min_split_gain=0.0, n_estimators=428,\n",
       "        n_jobs=4, num_class=9, num_leaves=50, objective='multiclass',\n",
       "        random_state=None, reg_alpha=0.0, reg_lambda=0.0, silent=False,\n",
       "        subsample=0.6, subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid=True, n_jobs=4,\n",
       "       param_grid={'colsample_bytree': [0.3, 0.4]},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':50,\n",
    "          'min_child_samples':30,\n",
    "          'max_bin': 127, \n",
    "          'subsample': 0.6,\n",
    "          'bagging_freq': 1,\n",
    "          #'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "colsample_bytree_s = [i/10.0 for i in range(3,5)]\n",
    "tuned_parameters = dict( colsample_bytree = colsample_bytree_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"neg_log_loss\", verbose=5, refit = False)\n",
    "grid_search.fit(X_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.47455852578860963\n",
      "{'colsample_bytree': 0.4}\n"
     ]
    }
   ],
   "source": [
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LGBMClassifier(bagging_freq=1, boosting_type='gbdt', class_weight=None,\n",
       "        colsample_bytree=0.4, importance_type='split', learning_rate=0.01,\n",
       "        max_bin=127, max_depth=7, min_child_samples=30,\n",
       "        min_child_weight=0.001, min_split_gain=0.0, n_estimators=100,\n",
       "        n_jobs=4, num_class=9, num_leaves=50, objective='multiclass',\n",
       "        random_state=None, reg_alpha=0.0, reg_lambda=0.0, silent=False,\n",
       "        subsample=0.6, subsample_for_bin=200000, subsample_freq=0)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#采用最佳参数重新训练模型\n",
    "arams = {'boosting_type': 'gbdt',\n",
    "          'objective': 'multiclass',\n",
    "          'num_class':9, \n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.01,\n",
    "          'n_estimators':n_estimators_2,\n",
    "          'max_depth': 7,\n",
    "          'num_leaves':75,\n",
    "          'min_child_samples':40,\n",
    "          'max_bin': 127, #2^6,原始特征为整数，很少超过100\n",
    "          'subsample': 0.8,\n",
    "          'bagging_freq': 1,\n",
    "          'colsample_bytree': 0.4,\n",
    "         }\n",
    "\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "lg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle as cPickle\n",
    "\n",
    "cPickle.dump(lg, open(\"Otto_LightGBM_org_tfidf.pkl\", 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame({\"columns\":list(feat_names), \"importance\":list(lg.feature_importances_.T)})\n",
    "df = df.sort_values(by=['importance'],ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>columns</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>159</th>\n",
       "      <td>feat_67_tfidf</td>\n",
       "      <td>1157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132</th>\n",
       "      <td>feat_40_tfidf</td>\n",
       "      <td>937</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117</th>\n",
       "      <td>feat_25_tfidf</td>\n",
       "      <td>925</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>152</th>\n",
       "      <td>feat_60_tfidf</td>\n",
       "      <td>924</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>feat_24_tfidf</td>\n",
       "      <td>909</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>feat_67</td>\n",
       "      <td>793</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>140</th>\n",
       "      <td>feat_48_tfidf</td>\n",
       "      <td>787</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>134</th>\n",
       "      <td>feat_42_tfidf</td>\n",
       "      <td>783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178</th>\n",
       "      <td>feat_86_tfidf</td>\n",
       "      <td>723</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>154</th>\n",
       "      <td>feat_62_tfidf</td>\n",
       "      <td>701</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>106</th>\n",
       "      <td>feat_14_tfidf</td>\n",
       "      <td>661</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>107</th>\n",
       "      <td>feat_15_tfidf</td>\n",
       "      <td>660</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>135</th>\n",
       "      <td>feat_43_tfidf</td>\n",
       "      <td>620</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>126</th>\n",
       "      <td>feat_34_tfidf</td>\n",
       "      <td>602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>feat_60</td>\n",
       "      <td>602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>180</th>\n",
       "      <td>feat_88_tfidf</td>\n",
       "      <td>592</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>feat_25</td>\n",
       "      <td>540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>feat_8_tfidf</td>\n",
       "      <td>504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>156</th>\n",
       "      <td>feat_64_tfidf</td>\n",
       "      <td>503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>feat_24</td>\n",
       "      <td>500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>feat_26_tfidf</td>\n",
       "      <td>497</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>feat_40</td>\n",
       "      <td>480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>feat_86</td>\n",
       "      <td>478</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>feat_48</td>\n",
       "      <td>472</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>feat_11_tfidf</td>\n",
       "      <td>471</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>164</th>\n",
       "      <td>feat_72_tfidf</td>\n",
       "      <td>468</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>163</th>\n",
       "      <td>feat_71_tfidf</td>\n",
       "      <td>466</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>108</th>\n",
       "      <td>feat_16_tfidf</td>\n",
       "      <td>449</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124</th>\n",
       "      <td>feat_32_tfidf</td>\n",
       "      <td>447</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>feat_43</td>\n",
       "      <td>447</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>feat_5_tfidf</td>\n",
       "      <td>46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138</th>\n",
       "      <td>feat_46_tfidf</td>\n",
       "      <td>46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>feat_10</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>feat_2</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>feat_2_tfidf</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>155</th>\n",
       "      <td>feat_63_tfidf</td>\n",
       "      <td>37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120</th>\n",
       "      <td>feat_28_tfidf</td>\n",
       "      <td>37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>144</th>\n",
       "      <td>feat_52_tfidf</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>153</th>\n",
       "      <td>feat_61_tfidf</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>113</th>\n",
       "      <td>feat_21_tfidf</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>feat_51_tfidf</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>feat_49</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>feat_84</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>feat_31_tfidf</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>feat_5</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>feat_46</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>feat_52</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>feat_31</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>feat_61</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>feat_63</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>feat_81</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>feat_51</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>174</th>\n",
       "      <td>feat_82_tfidf</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>feat_12</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>feat_6_tfidf</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>feat_21</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>feat_82</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>feat_65</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>feat_28</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>feat_6</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>186 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           columns  importance\n",
       "159  feat_67_tfidf        1157\n",
       "132  feat_40_tfidf         937\n",
       "117  feat_25_tfidf         925\n",
       "152  feat_60_tfidf         924\n",
       "116  feat_24_tfidf         909\n",
       "66         feat_67         793\n",
       "140  feat_48_tfidf         787\n",
       "134  feat_42_tfidf         783\n",
       "178  feat_86_tfidf         723\n",
       "154  feat_62_tfidf         701\n",
       "106  feat_14_tfidf         661\n",
       "107  feat_15_tfidf         660\n",
       "135  feat_43_tfidf         620\n",
       "126  feat_34_tfidf         602\n",
       "59         feat_60         602\n",
       "180  feat_88_tfidf         592\n",
       "24         feat_25         540\n",
       "100   feat_8_tfidf         504\n",
       "156  feat_64_tfidf         503\n",
       "23         feat_24         500\n",
       "118  feat_26_tfidf         497\n",
       "39         feat_40         480\n",
       "85         feat_86         478\n",
       "47         feat_48         472\n",
       "103  feat_11_tfidf         471\n",
       "164  feat_72_tfidf         468\n",
       "163  feat_71_tfidf         466\n",
       "108  feat_16_tfidf         449\n",
       "124  feat_32_tfidf         447\n",
       "42         feat_43         447\n",
       "..             ...         ...\n",
       "97    feat_5_tfidf          46\n",
       "138  feat_46_tfidf          46\n",
       "9          feat_10          41\n",
       "1           feat_2          39\n",
       "94    feat_2_tfidf          38\n",
       "155  feat_63_tfidf          37\n",
       "120  feat_28_tfidf          37\n",
       "144  feat_52_tfidf          35\n",
       "153  feat_61_tfidf          32\n",
       "113  feat_21_tfidf          32\n",
       "143  feat_51_tfidf          32\n",
       "48         feat_49          30\n",
       "83         feat_84          27\n",
       "123  feat_31_tfidf          23\n",
       "4           feat_5          22\n",
       "45         feat_46          21\n",
       "51         feat_52          21\n",
       "30         feat_31          18\n",
       "60         feat_61          17\n",
       "62         feat_63          14\n",
       "80         feat_81          13\n",
       "50         feat_51          11\n",
       "174  feat_82_tfidf          10\n",
       "11         feat_12          10\n",
       "98    feat_6_tfidf           8\n",
       "20         feat_21           7\n",
       "81         feat_82           7\n",
       "64         feat_65           6\n",
       "27         feat_28           5\n",
       "5           feat_6           1\n",
       "\n",
       "[186 rows x 2 columns]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#以下为测试\n",
    "import numpy as np\n",
    "test_data_path1=open(r'C:\\Users\\dell\\Downloads\\201913164947853_97888\\3代码类素材\\data\\Otto_FE_test_org.csv')\n",
    "test_data_path2=open(r'C:\\Users\\dell\\Downloads\\201913164947853_97888\\3代码类素材\\data\\Otto_FE_test_tfidf.csv')\n",
    "test1=pd.read_csv(test_data_path1)\n",
    "test2=pd.read_csv(test_data_path2)\n",
    "test2 = test2.drop([\"id\"], axis=1)\n",
    "test =  pd.concat([test1, test2], axis = 1, ignore_index=False)\n",
    "\n",
    "test_id = test['id']   \n",
    "X_test = test.drop([\"id\"], axis=1)\n",
    "feat_names = X_test.columns \n",
    "\n",
    "from scipy.sparse import csr_matrix\n",
    "X_test = csr_matrix(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#import cPickle\n",
    "model = cPickle.load(open(\"Otto_LightGBM_org_tfidf.pkl\", 'rb'))\n",
    "#输出每类的概率\n",
    "y_test_pred = model.predict_proba(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#生成提交结果\n",
    "out_df = pd.DataFrame(y_test_pred)\n",
    "\n",
    "columns = np.empty(9, dtype=object)\n",
    "for i in range(9):\n",
    "    columns[i] = 'Class_' + str(i+1)\n",
    "\n",
    "out_df.columns = columns\n",
    "\n",
    "out_df = pd.concat([test_id,out_df], axis = 1)\n",
    "out_df.to_csv(\"LightGBM_org_tfidf.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
